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Integrating  Software  and  Systems  Engineering 
to  Promote  Interoperability 


As  the  Acting  Director  of  Software  and  Systems  Engineering  in  the  Office  of  the 
Deputy  Under  Secretary  of  Defense  (Acquisition  and  Technology),  I  occupy  a 
uniquely  advantageous  position  to  witness  the  challenges  that  interoperability  imposes 
on  the  engineering  community  In  particular,  I  have  observed  that  very  distinct  cultures 
distinguish  the  software  engineering  and  systems  engineering  communities  and  the 
subtle  differences  in  their  perspectives,  and  how  these  have  undermined  the  DoD’s 
ability  to  develop  solutions  to  warfighter  needs.  Although  systems  engineering,  as  a  dis¬ 
cipline,  nominally  encompasses  software,  its  heritage  is  hardware-oriented  and  favors  a  product- 
oriented  perspective  and  functional  decomposition.  By  comparison,  the  software  community — 
with  roots  in  embedded  systems,  information  technology,  and  command  and  control  domains — 
embraces  layered  architectures  and  process-focused  development  perspectives. 

Perpetuation  of  functionally  stove-piped  policies  and  organizational  structures  has  permit¬ 
ted  each  community’s  worldview  to  somewhat  peacefully  co-exist  by  limiting  the  amount  of 
interaction.  Today’s  systems  cannot  ignore  the  need  to  interoperate.  Information  technology  has 
allowed  us  to  shift  the  balance  of  control  from  hardware  to  software  and  enabled  an  ad-hoc 
composition  of  systems  which  are  not  specifically  designed  to  interoperate.  We  face  warfighter 
demand  for  capabilities  that  are  joint  and  adaptable.  This  requires  acknowledgement  of  an 
enhanced  systems  engineering  imperative  to  seamlessly  integrate  hardware,  software,  and  human 
factors  and  enable  system  of  system  solutions.  We  must  synthesize  the  most  thoughtful  per¬ 
spectives  of  software  and  systems  engineering  to  capitalize  on  technology  and  deliver  integrat¬ 
ed  capabilities  to  our  customers. 

This  issue  of  CROSSTALK  addresses  some  of  these  compelling  challenges  as  we  strive  to 
improve  integration/interoperability.  In  Systems  Engineering  for  Capabilities,  Dr.  Judith  S.  Dahmann, 
George  Rebovich  Jr.,  and  Jo  Ann  Lane  adapt  systems  engineering  concepts  for  the  development 
of  capabilities.  Dr.  John  Colombi,  Maj.  Brannen  C.  Cohee,  and  Maj.  Chuck  W  Turner  discuss — 
in  Interoperability  Test  and  devaluation:  A  System  of  Systems  Yield  Study — the  policy  and  practice  of 
testing  for  interoperability  in  a  system  of  systems  context.  Shamlan  Siddiqi’s  Key  Transformational 
Techniques  to  Achieve  Enterprise-Scale  Interoperability  details  the  use  of  service-oriented  architecture 
design  principles  and  an  agile  methodology.  William  B.  Anderson  and  Philip  Boxer’s  Modeling  and 
Analysis  of  Interoperability  Risks  in  Systems  of  Systems  Environments  describes  how  their  techniques 
in  an  interoperability  risk  probe  found  gaps  in  the  ability  of  a  modernization  program  to  react 
to  changing  demands.  In  Quality  and  Cost—  Ids  Not  Either/  Or:  Making  the  Case  With  Cost  of  Quality , 
George  Webb  and  LTC  Nanette  Patton  describe  the  application  of  “cost  of  quality”  principles 
to  permit  the  balancing  of  these  attributes. 

I  cannot  overemphasize  the  importance  and  criticality  of  the  need  for  the  software  and  sys¬ 
tems  engineering  communities  to  jointly  confront  the  challenges  of  fielding  systems  that  are 
affordable,  sustainable,  and  interoperable. 


Kristin  Baldwin 

Acting  Director ;  Software  and  Systems  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense  (Acquisition  and  Technology j 
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Systems  Engineering  for  Capabilities 


Dr.  Judith  S.  Dahmann  and  George  Rebovich  Jr.  Jo  Ann  Lane 

The  MITRE  Corporation  University  of  Southern  California 

With  the  increased  emphasis  on  capabilities  and  networking,  the  DoD  is  recognising  the  criticality  of  effective  end-to-end 
performance  of  systems  of  systems  (SoS)  to  meet  user  needs.  While  acquisition  continues  to  focus  on  systems,  systems 
requirements  are  increasingly  based  on  assessment  of  gaps  in  user  capabilities  and  in  priority  areas;  there  is  an  increasing 
focus  on  integration  across  systems  to  enable  capabilities.  Thus,  the  role  of  systems  engineering  (SE)  is  expanding  to  the 
engineering  of  SoS  that  provide  user  capabilities.  This  article  discusses  the  shape  of  SoS  in  the  DoD  today.  It  outlines  a 
recent  initiative  to  provide  guidance  on  the  application  of  SE  processes  to  the  definition  and  evolution  of  SoS. 


Beginning  with  the  2000  Quadrennial 
Defense  Review  (QDR)  [1],  the  DoD 
has  been  reorienting  force  development 
processes  to  the  identification  and  sup¬ 
port  of  user  capabilities,  with  an  empha¬ 
sis  on  agile  composition  of  systems  to 
meet  a  range  of  changing  user  needs.  The 
Joint  Capabilities  Integration  and 
Development  System  [2]  was  created  in 
2003  to  identify  capability  needs  in  terms 
of  functional  concepts  and  validated 
material  needs  in  terms  of  capability 
gaps.  In  parallel,  the  DoD  5000  [3]  rec¬ 
ognized  capability  areas  and  spawned  ini¬ 
tial  efforts  to  develop  road  maps  for 
capabilities.  The  2006  QDR  [4]  continued 
this  trajectory  and,  based  on  institutional 
reform  and  governance  recommenda¬ 
tions,  the  DoD  has  created  capability 
portfolio  managers  in  a  further  effort  to 
view  investments  within  the  broader  con- 

Table  1 :  Types  of  SoS 


text  of  user  capabilities  [5]. 

At  the  same  time,  the  DoD  recog¬ 
nized  the  importance  of  SE  as  a  key 
enabler  of  systems  acquisition.  Policy 
emphasized  the  importance  of  SE  and 
renewed  emphasis  on  technical  planning 
and  authorities  [6,  7,  8],  bringing  atten¬ 
tion  to  the  robust  SE  in  systems  acquisi¬ 
tions.  More  recently,  the  SE  community 
has  recognized  the  need  for  discipline 
and  structure  in  the  engineering  of  SoS. 

The  Shape  of  SoS  in  the 
DoD  Today 

An  SoS  is  defined  as  a  set  or  arrangement 
of  systems  that  results  when  independent 
and  useful  systems  are  integrated  into  a 
larger  system  that  delivers  unique  capabil¬ 
ities  [9]. 

While  DoD  acquisition  largely  contin¬ 


ues  to  emphasize  development  of  indi¬ 
vidual  systems,  it  has  been  increasingly 
recognized  that  for  priority  capabilities  it 
is  important  to  provide  management  and 
SE  to  the  ensembles  of  systems  which 
work  together  to  support  user  capability 
needs.  From  a  networking  perspective,  a 
set  of  DoD  policies  have  been  promul¬ 
gated  with  the  objective  of  providing  the 
requisite  infrastructure  to  support  com¬ 
munications  and  data  exchange  among 
systems  [10,  11]. 

In  the  DoD  today,  there  are  several 
types  of  SoS  [12,  13],  as  shown  in  Table  1. 
The  U.S.  Army’s  Future  Combat  Systems 
is  the  best- known  example  of  directed  SoS. 
Communities  of  interest  are  good  exam¬ 
ples  of  DoD  collaborative  SoS ,  and  the 
Global  Information  Grid  is  the  predomi¬ 
nant  DoD  virtual  SoS. 

Increasingly,  the  DoD  is  facing  the 
challenges  of  acknowledged  SoS  by  recog¬ 
nizing  the  need  for  capability  manage¬ 
ment  and  SE  at  the  SoS  level  while  main¬ 
taining  the  management  and  technical 
autonomy  of  the  systems  contributing  to 
the  SoS  capability  objectives.  Examples 
of  this  type  of  SoS  are  the  Missile  De¬ 
fense  Agency’s  Ballistic  Missile  Defense 
System,  the  Air  Force’s  Air  Operations 
Center,  and  the  Naval  Integrated  Fire 
Control- Counter  Air  capability. 

In  the  DoD,  a  typical  strategy  for  pro¬ 
viding  end-to-end  support  for  new  capa¬ 
bility  needs  is  to  add  functionality  to  the 
inventory.  In  most  cases,  these  systems 
continue  to  be  needed  for  their  original 
requirements.  Consequently,  the  owner¬ 
ship  or  management  of  these  systems 
remains  unchanged  and  they  continue  to 
evolve  based  on  their  own  development 
and  requirements  processes  and  indepen¬ 
dent  funding. 

The  dual  levels  of  management, 
objectives,  and  funding  result  in  manage¬ 
ment  challenges  for  both  the  SoS  and  the 


Type 

Definition 

Virtual 

Virtual  SoS  lack  a  central  management  authority  and  a  centrally 
agreed-upon  purpose  for  the  system  of  systems.  Large-scale 
behavior  emerges — and  may  be  desirable — but  this  type  of  SoS 
must  rely  upon  relatively  invisible  mechanisms  to  maintain  it. 

Collaborative 

In  collaborative  SoS,  the  component  systems  interact  more  or  less 
voluntarily  to  fulfill  agreed-upon  central  purposes.  The  Internet  is  a 
collaborative  system.  The  Internet  Engineering  Task  Force  works 
out  standards  but  has  no  power  to  enforce  them.  The  central 
players  collectively  decide  how  to  provide  or  deny  service,  thereby 
providing  some  means  of  enforcing  and  maintaining  standards. 

Acknowledged 

Acknowledged  SoS  have  recognized  objectives,  a  designated 
manager,  and  resources  for  the  SoS;  however,  the  constituent 
systems  retain  their  independent  ownership,  objectives,  funding, 
as  well  as  development  and  sustainment  approaches.  Changes  in 
the  systems  are  based  on  collaboration  between  the  SoS  and  the 
system. 

Directed 

Directed  SoS  are  those  in  which  the  integrated  system  of  systems 
is  built  and  managed  to  fulfill  specific  purposes.  It  is  centrally 
managed  during  long-term  operation  to  continue  to  fulfill  those 
purposes  as  well  as  any  new  ones  the  system  owners  might  wish 
to  address.  The  component  systems  maintain  an  ability  to  operate 
independently,  but  their  normal  operational  mode  is  subordinated 
to  the  central  managed  purpose. 
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Aspect  of 
Environment 

System 

Acknowledged  SoS 

Management  and  Oversight 

Stakeholder 

Involvement 

Clearer  set  of  stakeholders. 

Stakeholders  at  both  system  level  and  SoS  levels  (including  the  system 
owners,  with  competing  interests  and  priorities);  in  some  cases,  the  system 
stakeholder  has  no  vested  interest  in  the  SoS;  all  stakeholders  may  not  be 
recognized. 

Governance 

Aligned  project  manager  and 
funding. 

Added  levels  of  complexity  due  to  management  and  funding  for  both  the  SoS 
and  individual  systems;  SoS  does  not  have  authority  over  all  the  systems. 

Operational  Environment 

Operational 

Focus 

Designed  and  developed  to  meet 
operational  objectives. 

Called  upon  to  meet  a  set  of  operational  objectives  using  systems  whose 
objectives  may  or  may  not  align  with  the  SoS  objectives. 

Implementation 

Acquisition 

Aligned  to  acquisition  categories 
milestones,  documented 
requirements,  SE  with  a 

SE  plan. 

Added  complexity  due  to  multiple  system  lifecycles  across  acquisition 
programs,  involving  legacy  systems,  developmental  systems,  new 
developments,  and  technology  insertion;  typically  have  stated  capability 
objectives  up  front  which  may  need  to  be  translated  into  formal  requirements. 

Test  and 
Evaluation 

Test  and  evaluation  of  the 
system  is  generally  possible. 

Testing  is  more  challenging  due  to  the  difficulty  of  synchronizing  across 
multiple  systems’  life  cycles,  given  the  complexity  of  all  the  moving  parts  and 
potential  for  unintended  consequences. 

Engineering  and  Design  Considerations 

Boundaries 

and 

Interfaces 

Focuses  on  boundaries  and 
interfaces  for  the  single  system. 

Focus  on  identifying  the  systems  that  contribute  to  the  SoS  objectives  and 
enabling  the  flow  of  data,  control,  and  functionality  across  the  SoS  while 
balancing  needs  of  the  systems. 

Performance 
and  Behavior 

Performance  of  the  system  to 
meet  specified  objectives. 

Performance  across  the  SoS  that  satisfies  SoS  user  capability  needs  while 
balancing  needs  of  the  systems. 

Table  2:  Comparison  of  Systems  and  SoS 


systems,  especially  when  their  objectives 
are  not  well- aligned.  In  turn,  these  man¬ 
agement  challenges  pose  technical  chal¬ 
lenges  for  the  systems  engineers,  espe¬ 
cially  the  SoS.  Table  2  (from  [14])  lists 
some  additional  observations  regarding 
the  differences  between  systems  and  SoS. 

Core  Elements  of  SE 
for  SoS 

To  support  SE  for  SoS,  the  Acquisition, 
Technology  and  Logistics  (AT&L)  Direc¬ 
torate  of  System  and  Software  Engi¬ 
neering  has  developed  the  “Systems 
Engineering  Guide  for  Systems  of 
Systems”  [14].  The  guide  is  based  on  the 
experiences  of  SE  practitioners  and 
researchers  currently  working  with  SoS.  It 
identifies  seven  core  elements  of  SoS  SE 
(as  shown  in  Figure  1)  along  with  their 
interrelationships. 

Using  these  core  elements  as  a  frame 
of  reference,  [14]  describes  how  the  16 
SE  processes  from  [9]  (described  in  Table 
3,  next  page)  are  applied  in  SoS.  As  sys¬ 
tems  engineers  support  SoS  SE,  they 
leverage  these  basic  engineering  process¬ 
es.  These  processes,  which  were  devel¬ 


oped  for  the  engineering  of  individual 
systems,  essentially  provide  a  set  of  tools 
for  the  systems  engineers  as  they  face  the 
challenges  of  SoS  engineering.  The 


nature  of  the  SoS  environment  affects 
the  way  these  processes  are  employed  to 
support  SoS  SE.  The  SE  processes  which 
apply  to  each  of  the  core  elements  are 
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Technical  Processes 

Requirements  Development  takes  all  inputs  from  relevant  stakeholders  and 
translates  the  inputs  into  technical  requirements. 

Logical  Analysis  is  the  process  of  obtaining  sets  of  logical  solutions  to  improve 
understanding  of  the  defined  requirements  and  the  relationships  among  the 
requirements  (e.g.,  functional,  behavioral,  temporal). 

Design  Solution  translates  the  outputs  of  the  Requirements  Development  and 

Logical  Analysis  processes  into  alternative  design  solutions  and  selects  a  final  design 
solution. 

Implementation  is  the  process  that  actually  yields  the  lowest-level  system  elements 
in  the  system  hierarchy.  The  system  element  is  made,  bought,  or  reused. 

Integration  is  the  process  of  incorporating  the  lower-level  system  elements  into  a 
higher-level  system  element  in  the  physical  architecture. 

Verification  confirms  that  the  system  element  meets  the  design-to  or  build-to 
specifications.  It  answers  the  question  “Did  you  build  it  right?” 

Validation  answers  the  question  of  “Did  you  build  the  right  thing?” 

Transition  is  the  process  applied  to  move  ...  the  end-item  system  to  the  user. 

Technical  Management  Processes 

Decision  Analysis  provides  the  basis  for  evaluating  and  selecting  alternatives  when 
decisions  need  to  be  made. 

Technical  Planning  ensures  that  the  SE  processes  are  applied  properly  throughout 
a  system’s  life  cycle. 

Technical  Assessment  measures  technical  progress  and  the  effectiveness  of  plans 
and  requirements. 

Requirements  Management  provides  traceability  back  to  user-defined  capabilities. 

Risk  Management  ensures  program  cost,  schedule,  and  performance  objectives 
are  achieved  at  every  stage  in  the  life  cycle  and  communicated  to  all  stakeholders 
the  process  for  uncovering,  determining  the  scope  of,  and  managing  program 
uncertainties. 

Configuration  Management  is  the  application  of  sound  business  practices  to 
establish  and  maintain  consistency  of  a  product’s  attributes  with  its  requirements  and 
product  configuration  information. 

Data  Management  addresses  the  handling  of  information  necessary  for  or 
associated  with  product  development  and  sustainment. 

Interface  Management  ensures  interface  definition  and  compliance  among  the 
elements  that  compose  the  system,  as  well  as  with  other  systems  with  which  the 
system  or  system  elements  must  interoperate. 

Table  3:  Technical  and  Technical  Management  Processes  [14] 


shown  in  Table  4  (from  [14]). 

The  following  sections  describe  these 
core  elements  of  SoS  SE  from  [9]. 

Translating  SoS  Capability 
Objectives  Into  High-Level  SoS 
Requirements  Over  Time 

When  an  SoS  is  first  acknowledged,  the 
SE  team  is  called  on  to  understand  and 
articulate  the  technical-level  expecta¬ 
tions  for  the  SoS.  SoS  objectives  are  typ¬ 
ically  couched  in  terms  of  needed  capa¬ 
bilities,  and  the  systems  engineer  is 
responsible  for  working  with  the  SoS 
manager  and  users  to  translate  these  into 


high-level  requirements  that  provide  the 
foundation  for  the  technical  planning  to 
evolve  the  capability  over  time.  To 
accomplish  this,  the  SoS  SE  team  needs 
to  understand  the  nature  and  the  dynam¬ 
ics  of  the  SoS  to  appreciate  both  the 
context  for  SoS  expectations  and  to 
anticipate  areas  of  the  SoS  that  are  most 
likely  to  vary  in  implementation  and 
change  over  time.  The  SoS  systems  engi¬ 
neer  has  a  continuous  active  role  in  this 
ongoing  process  of  translating  capabili¬ 
ty  needs  into  technical  requirements  and 
identifying  new  needs  as  the  situation 
changes  and  the  SoS  evolves. 


Understanding  the  Constituent 
Systems  and  Their  Relationships 
Over  Time 

A  key  SoS  SE  activity  involves  under¬ 
standing  the  systems  involved  in  provid¬ 
ing  the  needed  SoS  capabilities  and  their 
relationships  and  interdependencies  as 
part  of  the  SoS.  In  an  individual  system 
acquisition,  the  systems  engineer  is  typi¬ 
cally  able  to  clearly  establish  boundaries 
and  interfaces  for  a  new  system.  In  an 
SoS,  systems  engineers  must  gain  an 
understanding  of  the  ensemble  of  sys¬ 
tems  that  affect  the  SoS  capability  and 
the  way  they  interact  and  contribute  to 
the  capability  objectives.  Key  systems  can 
be  outside  of  the  direct  control  of  the 
SoS  management  but  still  have  large 
impacts  on  the  SoS  objectives,  and  it  may 
not  be  possible  to  identify  all  the  systems 
that  affect  SoS  objectives.  It  is  most 
important  to  understand  the  players  ass- 
sociated  with  key  systems,  their  relation¬ 
ships,  and  their  drivers  so  that  options  for 
addressing  SoS  objectives  can  be  identi¬ 
fied  and  evaluated,  and  impacts  of 
changes  over  time  can  be  anticipated  and 
addressed.  Understanding  the  functional¬ 
ity  of  each  system  is  the  basis  for  under¬ 
standing  (1)  how  the  systems  support  the 
SoS  objectives,  (2)  technical  details  of  the 
systems  pertinent  to  the  SoS  (e.g., 
approaches  to  sharing  or  exchanging  mis¬ 
sion  information),  and  (3)  the  current 
system  development  plans,  including  tim¬ 
ing  and  synchronization  considerations. 
Finally,  the  SoS  systems  engineer  needs  to 
identify  the  stakeholders  and  users  of 
SoS  and  systems,  and  understand  their 
organizational  context  as  a  foundation 
for  their  role  in  the  SoS  over  time. 

Assessing  the  Extent  to  Which  SoS 
Performance  Meets  Capability 
Objectives  Over  Time 

In  an  SoS  environment,  there  may  be  a 
variety  of  approaches  to  addressing  objec¬ 
tives.  This  means  that  the  systems  engi¬ 
neer  needs  to  establish  metrics  and  meth¬ 
ods  for  assessing  the  performance  of  the 
SoS  capabilities  which  are  independent  of 
alternative  implementation  approaches.  A 
part  of  effective  mission  capability  assess¬ 
ment  is  to  identify  the  most  important 
mission  threads  and  focus  the  assessment 
effort  on  end-to-end  performance.  Since 
SoS  often  comprises  fielded  suites  of  sys¬ 
tems,  feedback  on  SoS  performance  may 
be  based  on  operational  experience  and 
issues  arising  from  operational  settings.  By 
monitoring  performance  in  the  field  or  in 
exercise  settings,  systems  engineers  can 
proactively  identify  and  assess  areas  need¬ 
ing  attention,  emergent  behavior  in  the 
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Technical  Processes  Technical  Management  Processes 

SoS  Core  Elements 

Requirements 

Development 

Logical 

Analysis 

Design 

Solution 

Implementation 

Integration 

Verification 

Validation 

Transition 

Decision 

Analysis 

Tech  Planning 

Tech  Assess. 

Requirements 

Management 

Risk 

Management 

Configuration 

Management 

Data 

Management 

Interface 

Management 

Translating  Capability 
Objectives 

X 

X 

X 

X 

X 

Understanding  Systems  and 
Relationships 

X 

X 

X 

X 

X 

Assessing  Performance  to 
Capability  Objectives 

X 

X 

X 

X 

X 

Developing  and  Evolving  an 

SoS  Architecture 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Monitoring  and  Assessing 
Changes 

X 

X 

X 

X 

X 

Addressing  Requirements  and 
Solution  Options 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Orchestrating  Upgrades  to 

SoS 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Table  4:  Technical  and  Technical  Management  Processes  as  They  Apply  to  the  Core  Clements  of  SoS  SE 


SoS,  and  the  impacts  of  changes  in  con¬ 
stituent  systems  on  the  SoS. 

Developing,  Evolving,  and 
Maintaining  an  Architecture  for 
the  SoS 

Once  an  SoS  systems  engineer  has  clari¬ 
fied  the  high-level  technical  objectives  of 
the  SoS,  identified  the  systems  that  are  key 
to  SoS  objectives,  and  defined  the  current 
performance  of  the  SoS,  an  architecture 
overlay  for  the  SoS  is  developed,  begin¬ 
ning  with  the  existing  or  de  facto  architec¬ 
ture  of  the  SoS.  The  architecture  of  an 
SoS  addresses  the  concept  of  operations 
for  the  SoS  and  encompasses  the  func¬ 
tions,  relationships,  and  dependencies  of 
constituent  systems,  both  internal  and 
external.  This  includes  end-to-end  func¬ 
tionality  and  data  flow  as  well  as  commu¬ 
nications.  The  architecture  of  the  SoS 
provides  the  technical  framework  for 
assessing  changes  needed  in  systems  or 
other  options  for  addressing  require¬ 
ments.  In  the  case  of  a  new  system  devel¬ 
opment,  the  systems  engineer  can  begin 
with  a  fresh,  unencumbered  approach  to 
architecture.  However,  in  an  SoS,  the  sys¬ 
tems  contributing  to  the  SoS  objectives 
are  typically  in  place  when  the  SoS  is 
established,  and  the  SoS  systems  engineer 
needs  to  consider  the  current  state  and 
plans  of  the  individual  systems  as  impor¬ 
tant  factors  in  developing  an  architecture 
for  the  SoS.  In  developing  the  architec¬ 
ture,  the  systems  engineer  identifies 
options  and  trades  and  provides  feedback 


when  there  are  barriers  to  achieving  bal¬ 
ance  between  the  SoS  and  systems. 

Monitoring  and  Assessing  Potential 
Impacts  of  Changes  on  SoS 
Performance 

A  big  part  of  SoS  SE  is  anticipating 
change — outside  of  the  SoS  span  of  con¬ 
trol — that  will  impact  functionality  or 
performance.  This  includes  internal 
changes  in  the  constituent  systems  as  well 
as  external  demands  on  the  SoS.  Because 
an  SoS  contains  multiple  independent 
systems,  the  systems  engineer  must  be 
aware  that  these  systems  are  evolving 
independently  of  the  SoS,  possibly  in 
ways  that  could  affect  the  SoS.  By  under¬ 
standing  the  impact  of  proposed  or 
potential  changes,  the  SoS  systems  engi¬ 
neer  can  either  intervene  to  preclude 
problems  or  develop  strategies  to  miti¬ 
gate  the  impact  on  the  SoS. 

Addressing  SoS  Requirements  and 
Solution  Options 

An  SoS  has  requirements  both  at  the  level 
of  the  entity  formed  by  the  interoperating 
constituent  systems  and  at  the  level  of  the 
individual  systems  themselves.  Depending 
on  the  circumstances,  the  SoS  systems 
engineer  may  have  a  role  at  one  or  both 
levels.  At  the  SoS  level,  as  with  systems,  a 
process  is  needed  to  collect,  assess,  and 
prioritize  user  needs,  and  then  evaluate 
options  for  addressing  these  needs.  When 
identifying  viable  options  to  address  SoS 
needs,  it  is  key  for  the  systems  engineer  to 


understand  the  individual  systems  and 
their  technical  and  organizational  context 
and  constraints,  and  to  consider  the  impact 
of  these  options  at  the  systems  level.  It  is 
the  SoS  systems  engineer’s  role  to  work 
with  requirements  managers  for  the  indi¬ 
vidual  systems  to  identify  the  specific 
requirements  to  be  addressed  by  appropri¬ 
ate  systems  (i.e.,  to  collaboratively  derive, 
decompose,  and  allocate  requirements  to 
systems).  This  activity  is  compounded  at 
an  SoS  level  due  to  the  multiple  acquisition 
stakeholders  that  are  engaged  in  an  SoS. 
The  objective  is  to  identify  options  which 
balance  the  needs  of  the  systems  and  the 
SoS,  since  in  many  cases  there  may  be  no 
clear  decision  authority  across  the  SoS. 
Designs  for  implementing  changes  to  the 
systems  are  done  by  the  systems  engineers 
of  the  systems. 

Orchestrating  Upgrades  to  SoS 

Once  an  option  for  addressing  a  need  has 
been  selected,  it  is  the  SoS  systems  engi¬ 
neer’s  role  to  work  with  the  SoS  sponsor, 
the  SoS  manager,  as  well  as  the  con¬ 
stituent  systems’  sponsors,  managers,  sys¬ 
tems  engineers,  and  contractors  to  fund, 
plan,  contractually  enable,  facilitate,  inte¬ 
grate,  and  test  upgrades  to  the  SoS.  The 
actual  changes  are  made  by  the  consistent 
systems’  owners,  but  the  SoS  systems 
engineer  orchestrates  the  process.  The 
system  engineer  leads  the  synchroniza¬ 
tion,  integration,  and  test  across  the  SoS 
and  provides  oversight  to  ensure  that  the 
changes  agreed  to  by  the  systems  are 
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Assessing 
performance 
to  capability 
objectives 


Figure  2:  ‘Double  Vee’  Depiction  of  Upgrades  to 

implemented  in  a  way  that  supports  the 
SoS. 

Implementation  of  Orchestrating 
upgrades  to  SoS ,  along  with  the  elements 
Addressing  requirements  and  solution  options 
and  Assessing  performance  to  capability  objec¬ 
tives  can  be  viewed  as  an  extended  ver¬ 
sion  of  the  SE  Double  Vee  (see  Figure  2); 
the  SoS  systems  engineer  addresses  issues 
across  the  SoS  and  the  systems  engineers 
of  the  systems  address  changes  in  their 
systems. 

Summary 

Systems  engineers  are  increasingly  called 
upon  to  implement  SE  for  supporting 
user  capabilities  in  networked  environ¬ 
ments  and  are  charged  with  evolving  exist¬ 
ing  and  new  systems  to  meet  changing 
user  needs.  As  well,  they  are  challenged  to 
leverage  SE  processes  developed  and 
applied  for  SE  of  new  systems.  In  today’s 
SoS  environments,  individual  systems  are 
no  longer  considered  as  individual  bound¬ 
ed  entities,  but  rather  as  components  in 
larger  and  more  variable  ensembles  of 
interdependent  systems,  interacting  based 
on  end-to-end  business  processes  and  net¬ 
worked  information  exchange  to  meet 
user  capability  needs.  Because  they  are 
starting  with  existing  systems  with  inde¬ 
pendent  owners,  objectives,  and  develop¬ 
ment  processes,  systems  engineers  are 
faced  with  a  new  set  of  conditions  for 
their  SE  processes.  This  calls  for  a  new  SE 
framework,  reflecting  the  dynamics  and 


uncertainty  of  SoS  as  well  as  the  added 
complexity  of  operating  in  an  SoS  envi¬ 
ronment  to  meet  DoD  capability  needs. ♦ 
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Philip  Crosby  was  a  cost-of-quality  guru  whose  many  books 
included  the  groundbreaking  “Quality  Is  Free,”  which  is  still 
making  an  impact  well  after  its  silver  anniversary.  At  this  Web 
site,  learn  how  Crosby’s  book  has  impacted  others,  learn  its 
main  ideas,  and  get  a  history  lesson  on  the  career  and  impact  of 
the  man  Time  magazine  dubbed  “the  leading  evangelist  of  qual¬ 
ity  in  the  U.S.” 

Web  Services  (WS)  Specifications  and 
Service-Oriented  Architecture 
Interoperability 

http:// opensource.sys-con.com/ node/ 314083 
Just  following  WS  standards  and  guidelines  during  the  develop¬ 
ment  phase  of  a  project  isn’t  enough  to  achieve  interoperability 
This  article  from  Enterprise  Open  Source  magazine  provides  a  set 
of  guidelines,  insight,  and  best  practices  that  you  can  follow  to 
accomplish  interoperability  when  developing  WS  that  make  use 
of  specifications  across  products  provided  by  different  vendors, 
as  well  as  help  with  specifications  that  are  being  developed  by 
different  groups. 
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Interoperability  Test  and  Evaluation: 

A  System  of  Systems  Field  Study 

Dr.  John  Colombi,  Maj.  Brannen  C.  Cohee,  and  Maj.  Chuck  W  Turner 

Air  Force  Institute  of  Technology 

Effective  operational  test  and  evaluation  (OT&E)  is  an  essential  part  of  successful  systems  and  software  engineering.  But 
increased  program  dependencies,  network-centric  operations,  and  growing  interoperability  requirements  have  greatly  compli¬ 
cated  test  and  evaluation.  This  article  examines  the  policy,  process,  and  practice  of  the  Air  Force  (AT)  test  and  evaluation 
programs,  such  as  Force  Development  Evaluations  (FDEs),  particularly  during  the  sustainment  of  systems.  Several  obser¬ 
vations  are  made  regarding  the  current  process  and five  areas  are  emphasised for  improvement. 


An  increasing  challenge  is  facing  the 
OT&E  community  when  operating 
across  multiple  weapon  systems  at  various 
stages  of  development.  After  studying  the 
policy  process,  and  practice  associated 
with  an  AF  Major  Command’s  (MAJ- 
COM’s)  OT&E  program,  this  article  con¬ 
cludes  that  the  AF,  while  espousing  a  test¬ 
ing  philosophy  of  seamless  verification ,  still 
needs  to  transition  to  a  more  integrated 
system  of  systems  (SoS)  approach  to  plan¬ 
ning  and  executing  OT&E.  Too  often,  a 
fielding  decision  for  a  single  system  modi¬ 
fication  is  the  goal  for  smaller  test  events. 
This  system-centric  focus  can  be  mis¬ 
placed.  Indeed,  with  the  dawning  of  net¬ 
work-centric  operations,  the  SoS  impera¬ 
tive  is  even  greater  for  the  successful  inte¬ 
gration,  test,  and  evaluation  of  warfight¬ 
ing  capabilities.  This  article  highlights  sev¬ 
eral  observations  and  offers  some  areas 
for  process  improvement. 

To  confirm  these  challenges,  this 
analysis  focused  on  a  geographically  dis¬ 
persed  network  of  ground  stations  that 
work  with  AF  and  DoD  surveillance  and 
reconnaissance  (S&R)  platforms  to  pro¬ 
vide  data,  information,  and  knowledge 
services  for  the  joint  commander  and 
forces  in  the  field.  A  decade  ago,  these 
S&R  platforms  and  their  attending  ground 
stations  existed  in  isolation.  Since  then, 
however,  operational  necessities  and  tech¬ 
nological  opportunities  have  birthed  a  sys¬ 
tem  of  increasingly  interdependent  hard¬ 
ware  and  software  systems,  spanning  sen¬ 
sors,  platforms,  data  links,  communication 
networks,  and  software -intensive  ground 
processing  resources.  The  evolution  of 
this  SoS  has  produced  remarkable 
advances  in  the  war  fighting  capability,  but 
this  integration  has  also  created  a  host  of 
systems  engineering  (SE)  and  enterprise 
management  challenges,  such  as  OT&E 
planning  and  execution. 

The  SoS  Challenge 

The  very  nature  of  an  SoS  makes  the 
enterprise  management  of  traditionally 
system-centric  support  processes,  such  as 


OT&E,  difficult.  As  Mark  Maier  points 
out,  even  though  an  SoS  operates  syner- 
gistically,  systems  in  an  SoS  can  operate 
and  are  managed  independently  [1].  This 
is  a  premise  that  is  expected  to  continue 
into  the  foreseeable  future;  therefore,  a 
single  organization  or  program  manager 
will  not  suddenly  take  complete  manager¬ 
ial  control  of  all  systems  within  the  applic¬ 
able  SoS.  One  reason  is  that  a  system  may 
participate  in  multiple  mission  threads 
interacting  with  a  variety  of  joint  organi¬ 
zations  and  weapon  systems.  The 
“Systems  Engineering  Guide  for  Systems 
of  Systems”  recognizes  that  an  SoS  is  usu¬ 
ally  not  born  of  a  single  development 
effort  but  emerges  as  complex  combina¬ 
tions  of  newly  acquired  and  legacy  sys¬ 
tems — each  with  their  own  management, 
operations,  and  support  communities — 
that  evolve  over  time  [2].  Annette  Krygiel 
notes  that  the  purpose  and  capabilities  of 
an  SoS  change  as  functions  are  added, 
removed,  and  modified  [3] .  Compounding 
the  complexity  of  SoS  SE,  Pin  Chen  and 
Jennie  Clothier  observe  that  component 
systems  in  an  SoS  are  often  systems  of 
systems  themselves  [4].  All  of  this  compli¬ 
cates  the  current  test  and  evaluation 
approaches. 

Despite  these  challenges,  operational 
demands  are  forcing  the  AF  and  DoD  to 
co-evolve  historically  system-centric 
processes  like  OT&E  to  support  the 
development  of  SoS  and  net-centric  capa¬ 
bilities  [5].  Therefore,  to  better  understand 
the  need  for  and  obstacles  to  this  co-evo- 
lution,  this  study  focused  on  a  particular 
OT&E  process  and  the  extent  to  which  it 
supports  an  SoS  approach.  The  OT&E 
process  chosen,  called  an  FDE,  is  man¬ 
aged  at  the  MAJCOM  level  to  make  field¬ 
ing  decisions  for  operational  weapon  sys¬ 
tems  as  incremental  upgrades  are  made 
during  sustainment.  It  should  be  noted 
that  an  FDE  is  one  of  several  types  of 
OT&E  called  out  in  AF  Instruction  (AFT) 
99-3,  Capability  Based  Test  and 
Evaluation.  Others  include  Initial  Opera¬ 
tional  Test  and  Evaluation,  Qualification 


Operational  Test  and  Evaluation,  Follow- 
on  Operational  Test  and  Evaluation, 
Tactics  Development  and  Evaluation,  the 
Weapons  System  Evaluation  Program, 
Operational  Utility  Evaluation,  Opera¬ 
tional  Assessments,  and  Early  Operational 
Assessments. 

MAJCOMs  conduct  FDEs  for  pro¬ 
grams  requiring  full-rate  production  or 
fielding  decisions  if  the  AF  Operational 
Test  Center  chooses  not  to  conduct 
OT&E.  This  is  typically  true  for 
Acquisition  Category  III  programs  or 
maintenance  modifications.  After  a  system 
has  been  fielded  and  has  entered  the  sus¬ 
tainment  phase  of  its  life  cycle,  the  prima¬ 
ry  type  of  test  and  evaluation  used  to  ver¬ 
ify  and  validate  smaller  system  upgrades  is 
the  FDE.  As  stated  in  the  Air  Combat 
Command  instruction,  the  focus  of  FDE 
is  a  subset  of  OT&E.  FDEs  are  primarily 
concerned  with  sustainment,  pre-planned 
product  improvement,  as  well  as  tactics, 
techniques,  and  procedures  development. 
The  objective  is  to  demonstrate  the  oper¬ 
ational  effectiveness  and  suitability  of  a 
system  as  evolutionary  upgrades  are  made 
to  sustain  its  relevance  to  the  warfighter. 
In  the  Air  Combat  Command  (ACC)  FDE 
process,  ACC  Test  Centers  (e.g.,  the  AF 
Warfare  Center,  the  AF  Information 
Operations  Center,  and  the  Air  National 
Guard  AF  Reserve  Test  Center)  are 
responsible  for  planning  and  executing 
FDEs. 

Next,  the  OT&E  process  is  examined 
in  the  context  of  its  governing  law  and 
policy.  Then,  a  case  study  is  documented 
which  focuses  on  a  particular  test  event 
that  involved  a  networked  ground  system 
and  one  of  its  airborne  partners. 

Observations  From  the  Field 
Study 

From  Congress,  direction  for  OT&E 
flows  down  from  four  sections  of  Title  10 
of  the  U.S.  Code:  Director  of  OT&E; 
Survivability  Testing  and  Lethality  Testing 
Required  Before  Full-Scale  Production; 
OT&E  of  Defense  Acquisition  Programs; 
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System  Life  Cycle  Testing  (horizontal) 


DT&E  =  Developmental  Testing 

Figure  1 :  System  Integration  During  Interoperability  Testing 


and  Low- Rate  Initial  Production  of  New 
Systems.  The  DoD  then  implements  the 
policies  into  directives,  instructions,  and 
regulations.  The  AF  further  clarifies  its 
entire  test  and  evaluation  process  in  the 
99-series  of  departmental  instructions, 
such  as  AF  Policy  Directive  99-1,  Test  and 
Evaluation  Process,  and  AFI  99-103, 
Capabilities  Based  Test  and  Evaluation  [6]. 
This  policy  defines  the  purpose  of  the  AF 
test  and  evaluation  process  and  provides  a 
framework  for  test  activities.  It  also 
expands  on  the  two  major  types  of  tests: 
Developmental  Test  and  Evaluation  and 
OT&E.  The  AF  philosophy  clearly 
reflects  the  verification  and  validation  of 
mission-level  capabilities,  an  emphasis  on 
seamless  verification  across  the  developmen¬ 
tal  and  operational  test  activities,  the  use 
of  an  integrated  test  team  (ITT)  for  test 
management,  and  the  efficient  sharing  of 
test  data  through  a  common  database. 
Finally,  AF  MAJCOMs  define  operating 
procedures,  as  in  ACC  Instruction  (ACCI) 
99-101,  ACC  Test  and  Evaluation.  This 
instruction  further  clarifies  MAJCOM 
OT&E  procedures.  Two  examples  that 
stand  out  in  ACCI  99-101  are  the 
Electronic  Project  Order,  for  tasking  orga¬ 
nizations  and  resources,  and  the  use  of  a 
yearly  Test  Priority  List.  While  extensive, 
the  test  policy  hierarchy  is  observed  to 
provide  good  vision  and  insightful  guid¬ 
ance  and  establishes  a  foundation  for 
being  able  to  handle  today’s  SoS  test  chal¬ 
lenges.  Several  observations  on  the  cur¬ 
rent  process  are  provided  in  the  following 
sections. 

Seamless  Verification  Still  Has  Seams 

Through  our  policy  analysis,  it  was  discov¬ 
ered  that  the  AF  endorses  a  capabilities- 
based  concept  of  seamless  verification, 
especially  by  mandating  the  use  of  ITTs. 
The  ITT  consists  of  a  cross- functional 
group  of  empowered  representatives 
from  multiple  disciplines  and  organiza¬ 
tions  and  is  co-chaired  by  operational 
testers  and  the  program  manager.  The 
problem  lies  in  the  meaning  of  integrated. 
While  DoD  policy  includes  both  the  hori¬ 
zontal  integration  of  test  and  evaluation 
throughout  a  system’s  life  cycle  and  the 
vertical  integration  of  systems  under  inter¬ 
operability  testing  (as  shown  in  Figure  1), 
the  AF  sees  integrated  testing  almost 
exclusively  in  life-cycle  terms  [6] .  In  addi¬ 
tion,  AF  policy  also  mandates  the  use  of 
open,  shared  databases  for  managing  test 
information.  This  integrated  information 
management  could  be  how  the  acquisition 
and  test  communities  could  begin  to 
bridge  the  vertical  seams. 

Thus,  seamless  verification  still  has 


seams  separating  test  activities  among 
interdependent  weapons  systems.  AF  pol¬ 
icy  does  not  mandate  organizational  struc¬ 
tures  and  management  processes  that  help 
OT&E  organizations  conduct  their  testing 
activities  in  an  SoS  framework.  However, 
the  prevalence  of  systems  of  systems  and 
the  evolution  toward  net-centricity  de¬ 
mands  test  processes  that  are  both  inte¬ 
grated  throughout  the  life  cycle  of  a  single 
weapon  system  and  integrated  across 
entire  sets  of  operational  capability. 

SoS  Approach  Not  Built  In 

It  was  found  that  the  FDE  process  is  flex¬ 
ible  in  that  it  can  be  applied  to  both  sys¬ 
tem-centric  and  capabilities-based  SoS  test 
events.  Even  so,  it  does  not  include  steps 
to  intentionally  evaluate  SoS  capabilities 
rather  than  individual  systems.  Rather,  it 
relies  on  the  insight  and  foresight  of  the 
MAJCOM  staff  and  the  test  center  orga¬ 
nizations  to  properly  scope  events  to 
approximately  demonstrate  full  warfight¬ 
ing  capabilities. 

Indeed,  the  test  project  manager,  gen¬ 
erally  appointed  from  one  of  the 
MAJCOM’s  test  center  organizations,  is 
the  central  figure  in  defining  the  scope  of 
the  test.  Whether  determining  the  compo¬ 
sition  of  the  planning  team,  developing 
test  objectives,  requesting  additional  sup¬ 
port  for  testing,  or  developing  the  test 
plan,  the  extent  to  which  FDE  demon¬ 
strates  the  operational  capability  of  an  SoS 
depends  largely  on  the  vision  and  initiative 
of  the  individual  project  manager.  The 
MAJCOM’s  process  does  not  include 
functions  that  obligate  a  project  manager 
to  scope  a  test  at  the  SoS  level. 

Increasing  Load  on  OT&E 

The  studied  MAJCOM  has  experienced  a 
dramatic  increase  in  OT&E  requirements 
(from  around  200  just  five  years  ago  to 


around  300  today)  and  the  number  of 
short-notice  or  out-of-cycle  requirements 
(from  around  10  percent  of  the  total  num¬ 
ber  of  test  events  five  years  ago  to  approx¬ 
imately  40  percent  today).  The  war  on  ter¬ 
ror  has  certainly  contributed  to  both  the 
number  and  urgency  of  OT&E  require¬ 
ments,  but  one  subject  matter  expert  inter¬ 
viewed  believes  the  increased  number  of 
acquisition  spirals  and  increments  along 
with  the  introduction  of  non-traditional 
software-intensive  weapon  systems — such 
as  the  networked  ground  system  in  our 
case  study — has  led  to  a  more  expansive, 
dynamic,  and  complex  environment  for 
MAJCOM-led  testing. 

Experts  and  senior  leaders  have  argued 
that  the  growing  interdependence  of  sys¬ 
tems  organized  in  net-centric  architectures 
will  exacerbate  the  increased  load  (i.e.,  the 
number,  complexity,  tempo,  and  expense 
of  test  events),  calling  it  exponential 
growth.  With  testing  resources  either 
remaining  static  or  decreasing  over  time, 
this  increased  load  will  force  the  MAJ¬ 
COM — as  well  as  the  broader  test  com¬ 
munity — to  develop  new  methods  for  test¬ 
ing  and  evaluating  net-centric  capabilities. 

System-Centric  Approach  Breaks 
Down 

Our  case  study  starts  with  an  FDE  for  a 
modified  sensor  on  board  an  airborne 
platform.  This  sensor  modification  was 
needed  to  support  interoperability  with  a 
new  data  link  architecture,  and  it  included 
minor  software  upgrades  to  networked 
ground  stations.  The  MAJCOM  assigned 
the  event  to  its  organization  responsible 
for  testing  modifications  to  airborne  plat¬ 
forms.  Understanding  the  organic  rela¬ 
tionship  between  the  platform  and  the 
ground  system,  test  planners  knew  they 
needed  to  demonstrate  the  end-to-end 
interoperability  of  four  interdependent 
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systems:  the  sensor,  airborne  platform, 
data  link,  and  ground  station.  Yet,  the  test 
was  not  planned  as  a  demonstration  of  the 
operational  effectiveness  and  suitability  of 
an  SoS  capability,  but  as  a  validation  of  the 
single-sensor  system  with  the  support  of 
these  other  contributing  systems.  While 
this  may  seem  like  a  subtle  distinction,  it 
led  to  a  variety  of  coordination  and  com¬ 
munication  breakdowns  throughout  test 
planning.  In  retrospect,  the  FDE  and  sub¬ 
sequent  fielding  decision  should  have  been 
for  the  combined  SoS,  made  up  of  the  sen¬ 
sor,  airborne  platform,  data  link,  and 
ground  stations,  which  would  have  neces¬ 
sarily  involved  a  broader  cross-section  of 
stakeholders  from  the  genesis  of  the  test 
event.  When  SoS  thinking  is  not  baked  into 
the  overall  test  and  evaluation  process, 
even  highly  interdependent  systems  will 
have  difficulty  coordinating  test  events 
that  are  effective  and  relevant  for  the  con¬ 
stituent  systems  and  the  SoS  as  a  whole. 

Recommendations  for 
Net-Centric  OT&E 

Though  important  efforts  have  been 
made  at  all  levels  to  promote  SoS -level 
testing,  the  default  focus  of  testing  is  still 
on  individual  systems  as  opposed  to  whole 
capabilities.  As  net-centric  operations 
mature,  this  approach  will  have  to  change. 
Indeed,  as  our  field  study  indicates,  the 
DoD  is  already  feeling  the  pressure  that 
was  predicted  nine  years  ago: 

Testing  systems  will  become  far 
more  complex  since  the  focus  will 
not  be  on  the  performance  of  indi¬ 
vidual  systems,  but  on  the  perfor¬ 
mance  of  federations  of  systems.  [7] 

At  the  National  Defense  Industrial 
Association  Test  and  Evaluation  Summit 
in  2004,  DoD  officials  elaborated  on  the 
net-centric  challenges  to  traditional,  sys¬ 
tem-centric  testing  [8]: 

•  The  shifting  focus  from  platforms  to 
capabilities  and  SoS  solutions. 

•  The  increasing  complexity  of  systems 
combined  with  increasing  interdepen¬ 
dencies  among  systems  of  systems. 

•  The  increasing  operational  demand  for 
broader  and  deeper  integration  among 
disparate  systems. 

•  The  exponential  growth  in  functional 
and  physical  interfaces  introduced  by 
the  proliferation  of  network  partici¬ 
pants  (both  newly  developed  and 
legacy). 

•  The  increased  requirements  for  test 
and  evaluation  initiated  by  the  evolu¬ 
tionary  acquisition  philosophy  of 


build-a-little,  test-a-little. 

As  the  heralds  of  net-centricity 
emphasize,  the  DoD’s  transition  from 
Industrial  Age  (platform-centric)  to 
Information  Age  (net-centric)  operations 
must  include  a  co-evolution  of  supporting 
processes.  Testing  is  one  of  those  sup¬ 
porting  processes  that  must  co-evolve 
with  technology.  The  following  recom¬ 
mendations  are  believed  to  best  improve 
MAJCOM-level  OT&E,  but  should  be 
extensible  to  the  AF  and  DoD  OT&E. 
These  can  be  considered  attributes  of  a 
future  process;  while  not  meant  to  be 
exhaustive,  they  provide  a  starting  point 
for  continuing  research  on  how  to  evolve 
today’s  process  to  complement  a  more 
interoperable  net-centric  environment. 

Scope  OT&E  Events  at  the  SoS  Level 

This  article  advocates  an  SoS  (instead  of  a 
system-centric)  approach  to  OT&E. 
However,  the  question  arises  of  how  to 
appropriately  scope  the  boundaries  of  SoS 
test  events.  This  is  where  the  DoD  AF  and 
AF  Enterprise  Architecture  (EA)  could 
offer  practical  help.  As  the  use  of  EAs 
continues  maturing,  they  will  provide 
effective  models  for  assessing  how 
weapon  systems  can  and  should  interoper¬ 
ate  in  order  to  provide  warfighting  capa¬ 
bilities  to  the  joint  commander.  These 
models  will  help  planners  define  the 
boundaries  of  an  SoS,  and  they  will  docu¬ 
ment  what  the  MITRE  Corporation’s 
Prem  Jain  calls  a  mission  thread : 

A  precise,  objective,  description  of 
an  important  task  ...  a  time- 
ordered  operational  event  diagram 
that  captures  discrete,  definable, 
interactions  among  human  opera¬ 
tors  and/or  technological  compo¬ 
nents.  [9] 

Jain  argues  that  these  mission  threads  will 
support  modeling  and  simulation  (M&S) 
activities  for  net-centric  test  and  evalua¬ 
tion.  In  the  same  way  the  AF  uses  high- 
fidelity  simulators  to  slash  the  costs  asso¬ 
ciated  with  training  its  pilots,  the  test  com¬ 
munity  could  use  mission  thread-based 
M&S  to  validate  SoS  and  net-centric  capa¬ 
bilities  at  a  fraction  of  the  cost,  time,  and 
operational  impact  incurred  by  live,  end- 
to-end  tests. 

Validate  SoS  Interoperability 

By  advocating  SoS  testing,  an  endless  web 
of  end-to-end  interoperability  tests  is  not 
envisioned  with  every  other  possible 
weapon  system  and  every  configuration. 
Rather,  changes  to  individual  weapon  sys¬ 
tems  should  be  evaluated  according  to 


net-readiness  criteria  to  validate  their 
interoperability  with  the  rest  of  the  SoS  or 
net-centric  enterprise.  This  does  not  mean 
just  checking  to  see  if  a  modified  weapon 
system  is  IP-enabled.  Net-readiness  is  a 
comprehensive  concept  that  implies  inter¬ 
operability  at  many  layers  of  the  commu¬ 
nications  hierarchy  and  beyond:  physical, 
logical,  syntactic,  and  semantic  [2].  For 
nearly  30  years,  both  government  and 
industry  have  actively  explored  research 
on  interoperability  measurement  with  the 
goal  of  creating  a  straightforward  way  of 
measuring,  reporting,  and  then  improving 
the  interoperability  of  complex  networks 
of  people,  equipment,  processes,  and 
organizations.  Researchers  have  used 
more  than  30  definitions  of  interoperabil¬ 
ity  and  have  documented  more  than  60 
distinct  types  of  interoperability,  numer¬ 
ous  interoperability  attributes,  and  14 
foundational  interoperability  measure¬ 
ment  models  and  methodologies  [10]. 

For  SoS  testing,  a  set  of  net-readiness 
objectives  (see  Table  1)  based  on  the 
DoD’s  Net-Centric  Data  Strategy  [11]  is 
proposed.  Incorporating  these  objectives 
into  SoS  events  would  ensure  that  modi¬ 
fied  systems  continue  to  conform — at 
their  interface  with  the  network — to  the 
convergence  protocols  specified  in  the 
net-centric  architecture.  Instead  of  evalu¬ 
ating  all  end-to-end  relationships  to  vali¬ 
date  interoperability  within  the  SoS,  a  test 
event  could  confirm  the  integrity  of  the 
SoS  simply  by  demonstrating  the  modified 
system’s  adherence  to  the  network’s  con¬ 
vergence  protocols.  This  technique  would 
greatly  reduce  the  test  load  on  OT&E 
organizations  while  simultaneously  allow¬ 
ing  them  to  conduct  evaluations  focused 
on  the  operational  effectiveness  of  the 
net-centric  SoS,  as  opposed  to  the  individ¬ 
ual  systems  within  that  SoS. 

Prioritize  FDEs  According  to 
Operational  Risk 

Although  a  simulation  and  net-readiness 
demonstration  at  the  network  interface 
will  help  mitigate  the  test  load  associated 
with  SoS  and  net-centric  operations,  deci¬ 
sion  makers  will  still  expect  a  certain  level 
of  live,  end-to-end  testing  in  realistic  sce¬ 
narios  to  validate  higher-risk  capabilities. 
There  will  always  be  too  much  to  test, 
forcing  the  operational  and  test  communi¬ 
ties  to  develop  a  reasonable  means  of  pri¬ 
oritizing  test  events.  Complicating  this 
issue  is  a  fundamental  property  of  net- 
centric  operations:  New  transactions, 
interdependencies,  missions,  and  capabili¬ 
ties  as  additional  (even  unanticipated)  sen¬ 
sor,  shooter,  and  command  and  control 
nodes  join  the  network.  The  very  nature 


I  2  CROSSTALK  The  Journal  of  Defense  Software  Engineering 


November  2008 


Interoperability  Test  and  Evaluation:  A  System  of  Systems  Field  Study 


of  net-centric  operations  implies  that  the 
DoD  will  never  completely  anticipate  all 
of  the  relevant  operational  nodes  and  pre¬ 
cisely  how  those  nodes  will  interoperate  to 
accomplish  a  mission. 

Thus,  the  crucial  criterion  for  prioritiz¬ 
ing  and  scoping  SoS  OT&E  events  is 
operational  risk.  For  low-risk  development 
or  sustainment  efforts,  operational  deci¬ 
sion  makers  may  need  to  be  satisfied  with 
developmental  test  results  validated  in  an 
M&S-based  operational  test.  For  medium- 
risk  projects,  testers  may  use  a  synthetic 
test  strategy  that  employs  a  small  number 
of  distributed  operational  events  in  an 
M&S  framework.  The  test  and  evaluation 
community  may  need  to  reserve  tradition¬ 
al  end-to-end  events  for  the  highest  risk 
efforts.  The  key  will  be  for  operational 
decision  makers  to  set  an  appropriate  risk 
threshold  for  each  development  or  sus¬ 
tainment  program  and  seek  the  best  value 
test  option  in  terms  of  time,  money,  and 
testing/operational  resources  to  achieve 
that  threshold. 

Focus  on  Interfaces 

Without  trivializing  the  complexities  of  an 
SoS,  perhaps  more  emphasis  may  have  to 
be  placed  on  the  definition,  development, 
and  test  and  evaluation  of  interfaces. 
Interfaces  and  interface  management  have 
always  been  an  important  aspect  of  SE. 
Maier  and  Rechtin  state  this  design  heuris¬ 
tic:  “The  greatest  leverage  in  system  archi¬ 
tecting  is  at  the  interfaces  . . .  the  greatest 
dangers  are  also  at  the  interfaces”  [12]. 
According  to  the  “Defense  Acquisition 
Guidebook,”  this  heuristic  can  be  an  inter¬ 
face  that  is: 

The  [logical],  functional,  and  phys¬ 
ical  characteristics  required  to  exist 
at  a  common  boundary  or  connec¬ 
tion  between  persons,  between  sys¬ 
tems,  or  between  persons  and  sys¬ 
tems.  [13] 

During  an  interface  control  document 
(ICD)  review  of  one  space  program,  we 
examined  the  impact  of  interface  design 
and  development.  Over  a  three-year  peri¬ 
od  following  contract  award,  596  program 
engineering  items  were  examined.  These 
items  included:  requirements  changes, 
specification  updates  and  clarifications, 
ICD  changes,  SE  management  docu¬ 
ments,  baseline  schedule  changes,  test 
plans,  and  proposed  risk-mitigation 
actions;  ICD-related  actions  comprised 
190  (or  one-third)  of  the  total  number  of 
actions.  A  second  aspect  of  interface  man¬ 
agement  was  in  contract  costs.  From  a  set 
of  77  contractual  modifications  after  crit¬ 


Objective 

Description 

Visible 

Users  and  applications  can  discover  the  existence  of  data  assets 
through  catalogs,  registries,  and  other  search  services.  All  data 
assets  (intelligence,  non-intelligence,  raw,  and  processed)  are 
advertised  or  made  visible  by  providing  metadata,  which  describes 
the  asset. 

Accessible 

Users  and  applications  post  data  to  a  shared  space.  Posting  data 
implies  that:  (1 )  descriptive  information  about  the  asset  (metadata) 
has  been  provided  to  a  catalog  that  is  visible  to  the  enterprise  and 
(2)  the  data  is  stored  such  that  users  and  applications  in  the 
enterprise  can  access  it.  Data  assets  are  made  available  to  any 
user  or  application  except  when  limited  by  policy,  regulation,  or 
security. 

Understandable 

Users  and  applications  can  comprehend  the  data,  both  structurally 
and  semantically,  and  readily  determine  how  the  data  may  be  used 
for  their  specific  needs. 

Trusted 

Users  and  applications  can  determine  and  assess  the  authority  of 
the  source  because  the  pedigree,  security  level,  and  access 
control  level  of  each  data  asset  is  known  and  available. 

Agile 

Many-to-many  exchanges  of  data  occur  between  systems,  through 
interfaces  that  are  sometimes  predefined  or  sometimes 
unanticipated.  Metadata  is  available  to  allow  mediation  or 
translation  of  data  between  interfaces,  as  needed. 

Responsive 

Perspectives  of  users,  whether  data  consumers  or  data  producers, 
are  incorporated  into  data  approaches  via  continual  feedback  to 
ensure  satisfaction. 

Table  1 :  A  Template  for  Net-Readiness  Testing  Objectives 


ical  design  review,  43  (nearly  56  percent) 
were  in  some  way  related  to  interfaces 
either  through  studies,  ICD  updates,  or 
implementations  of  necessary  require¬ 
ment  changes.  More  interestingly,  ICD- 
related  issues  resulted  in  nearly  $31.5  mil¬ 
lion  (or  44  percent)  of  the  cost  impact  to 
this  program.  Although  this  is  just  one 
example,  it  is  an  indication  of  interface 
management  challenges  during  develop¬ 
ment  and  clearly  represent  an  area  that  will 
require  similar  effort  during  OT&E. 

Unfortunately,  interface  management 
alone  may  be  insufficient  to  understand 
the  complexity  within  an  SoS.  For  net¬ 
work-centric  information  systems,  one 
must  examine  the  internal  properties  and 
behaviors  of  the  logically  connected  sys¬ 
tems  and  their  users  who  find,  fuse,  mod¬ 
ify,  and  ultimately  use  shared  data. 

Employ  Integration  Environments 

The  heavy  use  of  M&S  and  synthetic  test¬ 
ing  to  supplement  traditional  test  and  eval¬ 
uation  presupposes  the  use  of  integration 
environments  to  build  SoS  and  net-centric 
operational  capabilities.  Annette  Krygiel 
calls  an  integration  environment: 

...  a  concept,  not  an  organization. 

It  is  the  environment  of  people, 
processes,  and  infrastructure  used 
by  a  team  consisting  of  acquisition 
and  operational  personnel  to  man¬ 
age  the  integration  before  the 
product  is  deployed  for  an  opera¬ 


tion  or  an  experiment  and  to  sus¬ 
tain  it  afterward.  [3] 

Thus,  an  integration  environment  is  a 
concept  that  transcends  not  just  OT&E 
but  is  an  essential  SE  infrastructure  for  the 
early  and  continuous  integration  of  testing 
efforts  in  SoS  development  and  sustain¬ 
ment.  Thus,  an  integration  environment  is 
critical  in  squeezing  the  most  overall  value 
from  OT&E  and  in  reducing  the  amount 
of  effort  required  late  in  the  development 
cycle  [14]. 

Clearly,  integrated  databases  open  to 
all  test  and  evaluation  stakeholders  are  just 
the  tip  of  the  iceberg  in  terms  of  the  tools 
needed  to  achieve  seamless  verification. 
Employing  integration  environments 
would  allow  test  teams  to  achieve  synergy 
throughout  the  life  cycle  of  component 
systems  and  across  the  networked  SoS.  It 
would  allow  early,  comprehensive,  and 
ubiquitous  test  and  evaluation  throughout 
the  development  of  SoS  capabilities, 
reducing  the  test  load  on  test  center  orga¬ 
nizations  and  giving  them  the  freedom  to 
focus  on  adding  value  where  it  counts  for 
MAJCOM  decision  makers:  in  reducing 
the  operational  risk  of  fielding  and 
employing  warfighting  capabilities. 

Summary 

This  field  study  of  the  policy,  process,  and 
practice  of  FDE  concludes  that  the  testing 
community  must  shift  from  system-centric 
testing  to  a  more  SoS  approach.  The 
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DoD’s  ongoing  transformation  to  net-cen¬ 
tric  operations  makes  the  co-evolution  of 
SoS  test  and  evaluation  an  even  greater 
imperative.  While  the  five  areas  for 
improvement  were  derived  from  MAJ- 
COM  FDE  practice,  they  reflect  similar 
concepts  for  more  formal  OT&E. 
Improving  the  operational  realism  of  net¬ 
work-centric  environments,  ensuring  time¬ 
ly  performance  of  operational  informa¬ 
tion,  and  facilitating  adequate  test  and  eval¬ 
uation  resources  continue  to  be  priorities 
of  the  OT&E  director  [15].  Likewise,  AF 
OT&E  continues  to  be  challenged  with 
SoS  test  planning  and  execution.  The  soft¬ 
ware  engineering  community  must  contin¬ 
ue  to  design  and  test  capabilities  with  an 
SoS  focus  and  develop  advanced  modeling 
and  simulation  capabilities  to  enable 
affordable  SoS  testing  in  a  net-centric  envi¬ 
ronment.  Likewise,  the  test  community, 
while  emphasizing  seamless  verification, 
needs  to  make  continued  progress  in  capa- 
bilities-based  interoperability  testing  across 
information-intensive  SoS.^ 
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Key  Transformational  Techniques  to 
Achieve  Enterprise-Scale  Interoperability 
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This  article  examines  key  modernisation  and  transformation  strategies  for  interoperability,  including  enterprise  use  of  open 
source,  service-oriented  architecture  (SOAJ,  and  agile  techniques  in  software  development.  The  article  concludes  with  a  real-world 
case  study  on  legacy  modernisation  and  interoperability  for  a  major  government  agency  through  use  of  these  tools  and  techniques. 


Modem  concepts  such  as  SOA  and 
principles  around  Web  2.0  and  Web 
3.0  rely  on  interoperability  as  their  foun¬ 
dation.  This  is  different  from  the  tradi¬ 
tional  view  of  integration  where  one  or 
more  adapters  are  required  to  glue  things 
together.  The  intelligent  use  of  standard 
interfaces  between  multiple  components, 
leveraging  open  source  technologies, 
sophisticated  SOA  design  principles,  and  a 
flexible  software  development  methodol¬ 
ogy  can  be  a  much  more  cost-effective, 
efficient,  and  scalable  option  for  the  enter¬ 
prise  interoperability  of  systems. 

While  the  adoption  of  open  source 
technologies  for  interoperability  has  been 
growing  at  a  fast  pace  over  the  last  few 
years,  I  find  that  there  is  still  apprehension 
in  the  enterprise  use  of  open  source  as  the 
dominant  platform  in  the  interoperability 
of  mission-critical  systems.  By  enterprise  use , 
I  mean  using  a  complete  stack  of  non-pro¬ 
prietary  open  source  tools  for  everything 
including  a  back-end  database  at  the  data 
layer,  core  applications  and  modules  at  the 
application  layer,  a  messaging  engine/ser- 
vice  bus  or  standard  interface  strategy  at 
the  integration  layer,  and  creative  user 
interface  design  techniques  leveraging 
advances  in  the  Web  2.0  space.  That  being 
said,  open  source  software  by  its  very 
nature  requires  that  its  software  code  be 
open,  extensible,  and  openly  available. 
This  paradigm  enables  programmers  glob¬ 
ally  to  share  thoughts  and  ideas,  constantly 
increasing  the  power  and  quality  of  devel¬ 
opment  tools  and  large  applications. 
Additionally,  I  believe  that  the  rapid  adop¬ 
tion  of  SOA  design  principles  is  a  signifi¬ 
cant  boost  to  the  modernisation  of  interop¬ 
erable  open  source  tools.  Of  course,  this 
rapid  evolution  of  sorts  creates  problems. 

Issues  With  Interoperability 
and  Security  in  the  Open 
Source  Arena 

With  new  tools  and  technologies  being 
released  often,  agencies  face  challenges  in 
determining  whether  a  particular  tool  will 
work  with  an  existing  legacy  component, 
whether  the  operating  system  is  compati¬ 


ble,  and  if  the  database  will  work.  There 
are  also  the  myriad  complexities  that  come 
with  defining  the  proper  interface  to  cre¬ 
ate  reusability.  Some  major  interoperabili¬ 
ty  issues  include  centralized  identity  man¬ 
agement,  data  integration  including  real¬ 
time  data  synchronization,  batch  transfer, 
and  portability.  This  is  critical  as  organiza¬ 
tions  want  their  solutions  to  work  across 
different  platforms,  such  as  the  various 
Linux  options  as  well  as  Windows. 

Other  issues  include  the  perception  of 
security  (or  lack  thereof)  with  open  source 
tools  as  well  as  the  trustworthiness  of  the 

“This  paradigm 
enables  programmers 
globally  to  share 
thoughts  and  ideas, 
constantly  increasing  the 
power  and  quality  of 
development  tools  and 
large  applications.” 

data.  It  is  easy  for  developers  to  be  able  to 
add  malicious  code  or  add  functions  (etc.) 
to  source  code  they  have  downloaded. 
Both  of  these  challenges  can  be  mitigated 
by  the  appropriate  selection  of  open 
source  projects.  It  is  advisable  to  select 
projects  that  are  more  established,  come 
from  a  reliable  source,  and  have  built-in 
security  mechanisms  rather  than  ones  that 
may  not  have  huge  support  or  documen¬ 
tation.  Typically,  source  code  from  reliable 
projects  does  not  usually  have  too  many 
security  issues  because  they  are  controlled 
by  an  open  source  committee  or  group. 
Many  development  projects  use  open 
source  tools  and  technologies  and  provide 
ongoing  maintenance  and  support  on 
those  projects  as  well.  In  fact,  based  on 
cases  I  have  encountered,  support  and 
maintenance  issues  are  resolved  more 


rapidly  with  open  source  because  a  devel¬ 
oper  can  go  into  the  code  and  make  the 
fix.  The  alternative — waiting  for  a  product 
vendor  to  come  in,  assess  the  problem, 
and  make  adjustments — usually  takes  a 
significant  amount  of  valuable  develop¬ 
ment  time  and  money. 

Interoperability  Using  an  SOA 
Design  and  Enabled  by  Web 
Services  (WS)  and  theWS* 
Protocols 

Using  the  SOA  design  paradigm,  interop¬ 
erability  is  typically  accomplished  by 
developing  WS  using  industry  standard 
programming  languages  such  as  extensi¬ 
ble  Markup  Language  (XML),  Web 
Services  Description  Language  (WSDL), 
and  others.  These  basic  WS  standards 
have  evolved  over  the  last  few  years  and 
the  Web  Services  Interoperability 
Organization  (WS-I) — an  open  industry 
organization  chartered  to  establish  best 
practices  for  WS  interoperability — has 
released  Basic  Profile  (BP),  containing 
implementation  guidelines  for  basic  WS 
standards.  Many  open  source  tools  are 
starting  to  adopt  these  standards  using 
guidance  from  BP  as  part  of  their  ongoing 
product  road  map.  I  think  this  is  a  critical 
success  criterion  in  the  further  evolution 
and  adoption  of  open  source  tools  for 
enterprise  integration  and  other  enter¬ 
prise-level  functions.  However,  just  fol¬ 
lowing  WS-I  standards  and  guidelines  is 
not  sufficient  enough  to  achieve  interop¬ 
erability. 

Best  Practices  in 
Implementing  WS-Enabled 
SOA  Interoperability 

It  is  important  to  note  that  the  WS*-based 
development  can  be  complex.  Based  on 
experience  (and  as  typically  in  many 
things),  an  incremental  strategy  is  the 
most  effective  one.  Start  implementing 
using  only  the  basic  WS  specification  as 
opposed  to  multiple  WS*  protocols,  which 
can  add  more  complexity  to  development. 
This  will  create  problems  when  trou- 
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ble shooting.  Many  government  organiza¬ 
tions  are  facing  major  development  hur¬ 
dles  because  of  large-scale  implementa¬ 
tions  of  WS  without  using  a  chunking 
strategy.  These  days,  vendors  provide  BP- 
compliant  products  and  have  tested  those 
products  comprehensively  for  interoper¬ 
ability.  It  is  important  to  follow  these 
guidelines  and  use  these  tools.  The  open 
source  community  (as  previously  men¬ 
tioned)  is  adopting  these  specifications  for 
interoperability  as  well.  Additionally,  it  is 
always  important  to  seek  advice  from  mul¬ 
tiple  forums  and  blogs,  as  there  are  people 
who  may  have  encountered  similar  issues. 

The  BP  consists  of  guidelines  or  best 
practices  recommending  how  the  men¬ 
tioned  specifications  (XML,  WSDL,  etc.) 
should  be  used  together  to  develop  inter¬ 
operable  WS.  The  guidelines  cover  mes¬ 
saging,  description,  discovery,  and  security. 
Some  key  Web  service  specifications  cov¬ 
ered  in  the  initial  release  (BP  1.0)  include 
common  standards  such  as  XML  1.0  and 
WSDL  l.l1. 

When  discussing  the  importance  of 
interoperability  and  standards,  I  like  using 
the  example  of  Boeing.  When  taking  apart 
old  747s,  Boeing  reuses  a  lot  of  the  core 
parts  (recovering  more  than  $6.8  million) 
due  to  the  use  of  common  standards  in 
the  design  of  the  original  Boeing  747. 
However,  specific  standards  must  be 
agreed  upon  before  this  reuse  can  take 
place,  and  this  is  not  applicable  to  other 
jets.  Similarly,  for  two  applications  to  com¬ 
municate,  they  must  also  agree  on  the  spe¬ 
cific  data  standards  to  be  encoded,  such  as 
using  XML;  this,  unfortunately,  does  not 


always  happen  due  to  conflicting  specifi¬ 
cations  adopted  by  different  vendors  and 
agencies.  A  case  in  point  is  the  proposed 
reliable-network  protocol  that  identifies, 
manages,  and  tracks  the  reliable  delivery  of 
messages  between  source  and  destination 
WS.  According  to  [1],  there  are  currently 
two  competing  specifications:  WS-Reliable 
Messaging  (using  a  specification  support¬ 
ed  by  IBM,  Microsoft,  BEA,  and  TIBCO) 
versus  WS-Reliability  (promoted  by 
Oracle,  Sun,  Fujitsu,  Hitachi,  and  Sonic 
Software,  among  others).  The  open  source 
world  is  starting  to  adopt  these  standards 
feverishly  in  the  hopes  of  expanding  their 
imminent  presence  in  the  larger  software 
market. 

Case  Study 

The  power  of  these  open  source  tools  and 
the  low  cost  of  development  are  precisely 
why  Keane,  a  billion-dollar  global  business 
and  IT  consulting  firm,  went  with  a  pre¬ 
dominantly  open  source  model  for  the 
design  and  development  of  the  USAF 
Logistics  Systems. 

Through  both  technology  and  business 
insight,  the  project  has  been  successful  in 
its  use  of  cost-effective  interoperable  open 
source  tools  to  help  the  USAF  achieve  its 
mission  and  goals.  The  system  handles 
5,000  users  and  around  3,000  simultaneous 
users  during  peak  time.  The  project  is  sav¬ 
ing  around  2  million  lines  of  code  by  using 
a  composite  commercial  off-the-shelf 
approach  and  utilizing  an  intelligent  mix  of 
open  source  tools.  This  cost-effective 
combination  provides  the  USAF  with  the 


interoperability  and  visibility  of  assets 
across  both  retail  and  wholesale  systems.  It 
also  integrates  the  information  in  near 
real-time  into  an  intuitive,  Web-based 
interface  using  Google  Web  Tools  to 
enhance  the  Web  2.0  capability.  The  pro¬ 
ject  also  used  an  SOA-based  approach  to 
integration,  leveraging  the  latest  in  open 
source  libraries  and  adopting  standards 
such  as  XML  and  the  Java  platform;  these 
rapidly  delivered  interoperability  with 
numerous  systems  and  exposed  legacy  sys¬ 
tems  without  modification. 

Some  core  and  notable  open  source 
technologies  used  included  the  Spring 
Framework,  an  application  and  integration 
framework  for  the  Java  platform;  iBatis 
for  persistence;  Apache  Axis,  an  XML- 
based  Web  service  framework  application 
development  done  in  Java  using  Tomcat 
and  extensive  use  of  open  source  libraries; 
and  Java  Enterprise  Edition/Java  2 
Enterprise  Edition  components.  Using 
these  technologies  to  develop  an  innova¬ 
tive  transformation  and  business  rules 
engine  is  an  intelligent  enterprise  use  of 
open  source  technologies  to  achieve  inter¬ 
operability. 

Enterprise  Software 
Development  Methodology  - 
the  Blended  Agile-Rational 
Unified  Process 

For  the  USAF  Logistics  Systems,  a  blend¬ 
ed  approach  to  enterprise  application 
development  combining  agile  and 
VIEWW2  methodologies  was  used  to 
quickly  deliver  modernized  solutions 
meeting  100  percent  of  client  expecta¬ 
tions.  This  collaborative  development 
approach  allowed  the  team  to  more  effi¬ 
ciently  review  and  test,  thereby  helping  to 
speed  development  efforts.  The  Agile- 
Rational  Unified  Process  (RUP)  method¬ 
ology  is  beneficial  because  it  works  with 
poorly  understood  architectures,  produces 
a  highly  reliable  system,  produces  a  system 
with  large  growth  capability,  manages 
risks,  can  be  constrained  to  a  predefined 
schedule,  provides  management  with 
progress  visibility,  and  allows  for  in- 
process  corrections.  Figure  1  depicts  the 
Agile-RUP  methodology  in  some  detail. 

Through  utilizing  open  source  tools 
and  innovative  forward-thinking  solutions, 
government  best  practices  can  be  used  as 
examples  to  follow  in  the  commercial 
arena.  Served  industries — such  as  defense 
and  aerospace,  pharmaceutical,  manufac¬ 
turing,  and  financial — could  benefit  great¬ 
ly  from  efficient  and  cost-effective  enter¬ 
prise-level  supply  chain  systems,  collabo¬ 
rative  decision  support  engines,  and  intel- 
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ligent  rules  agents  that  change  as  the  glob¬ 
al  economy  changes. 

Conclusion 

I  believe  that  the  adoption  of  interopera¬ 
ble  open  source  tools  using  SOA  design 
principles  will  drive  the  future  develop¬ 
ment  of  innovative  forward-thinking  solu¬ 
tions.  Government  best  practices  will  be 
used  as  examples  to  follow  in  the  com¬ 
mercial  arena.  Agencies  supporting 
defense  and  intelligence — as  well  as  the 
broader  federal  health  care  and  financial 
arenas — will  benefit  greatly  from  efficient 
and  cost-effective  interoperable  systems  at 
the  enterpris e-level.  ♦ 

Notes 

1.  Learn  more  about  BP  1.0  at  WS-I: 
<www.ws-i.org/>. 

2.  VIEWW  stands  for  the  different  phas¬ 
es  of  the  development  life  cycle: 
Vision,  Inception,  Elaboration,  Work, 
and  Web.  It  is  Keane’s  equivalent  to 
the  RUP. 
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Modeling  and  Analysis  of  Interoperability 
Risk  in  Systems  of  Systems  Environments 

William  B.  Anderson  and  Philip  Boxer 

Software  Engineering  Institute 

This  article  describes  the  use  of  a  set  of  modeling  and  analysis  techniques  in  an  interoperability  risk  probe  that found  gaps 
in  the  ability  of  a  North  Atlantic  Treaty  Organisation  (NATO )  modernisation  program  to  react  to  changing  demands. 

The  modeling  and  analysis  techniques  were  used  to  create  models  of  the  people ,  processes ',  and  technologies  of  the  program 
and  to  represent  the  way  demands  were  placed  on  this  complex  socio-technical  system.  Analysis  of  the  models  revealed  inter¬ 
operability  risks  that  were  manifested  in  the  linkages  between  operational  requirements  of  functional  capabilities  and  the 
way  in  which  those  capabilities  were  being  maintained.  The  risks  identified  in  this  probe  were  typed  as  mission,  composi¬ 
tion,  and  performance  risks.  The  structural  models  produced  by  the  techniques  bring  a  welcome  engineering  rigor  to  the 
process  of  examining  interoperability. 


A  National  Defense  Industrial  Associa¬ 
tion  report  suggests  that  modeling 
can  aid  the  DoD  throughout  the  system  of 
systems  (SoS)  development  life  cycle  [1]. 
Model-based  dynamic  system  analysis  pro¬ 
vides  insight  into  otherwise  unobservable 
dimensions  that  can  help  characterize  an 
SoS.  One  of  those  dimensions  is  interop¬ 
erability  risk,  which  is  manifested  in  the 
linkages  between  operational  require¬ 
ments  of  functional  capabilities  and  the 
way  in  which  those  capabilities  are  main¬ 
tained. 

This  article  describes  a  model-based 
interoperability  risk  probe1  of  a  NATO 
modernization  program.  Using  a  rapid 
assessment  engagement  format,  the 
Software  Engineering  Institute  (SEI) 
modeled  the  NATO  program  as  a  system 
of  social  and  technical  systems.  The  probe 
involved  workshops  and  interviews  con¬ 
ducted  over  a  two-week  period,  followed 
by  analysis  of  the  data  gathered.  In  the 
probe,  we  interpreted  SoS  and  interoper¬ 
ability  in  a  broad  sense.  We  examined  the 
hardware  and  software  in  the  context  of 
its  operational  and  sustainment  environ¬ 
ments.  Therefore,  the  SoS  examination 
included  the  many  ground  and  airborne 
systems  and  the  diverse  organizations 
(social  systems)  required  to  operate  and 
sustain  the  NATO  program.  In  this  arti¬ 
cle,  we  are  emphasizing  the  modeling  and 
analysis  techniques  employed  over  the 
specific  details  of  the  case,  because  those 
details  are  confidential  to  NATO2. 


The  risk  probe  emphasized  the  impor¬ 
tance  of  demand  on  a  systems  of  systems 
environment.  If  demand  is  stable  and  pre¬ 
identified — -large  nation-state  military  threat 
scenarios,  a  huge  and  stable  demand  for 
sport  utility  vehicles,  or  the  best  health  care 
that  money  can  buy — traditional  hierarchi- 

“If  demand  is  stable 
and  pre-identified ... 
traditional  hierarchical 
structures  and  monolithic 
systems  work  well. 
However,  conditions 
changed ...  forcing 
market-driven  demand 
responses  from  systems 
of  systems.” 

cal  structures  and  monolithic  systems  work 
well.  However,  demand  conditions  can 
change — terrorism  has  redefined  military 
threats,  ever-increasing  gasoline  prices  have 
affected  consumer  choice  in  automotive 
vehicles,  and  health  care  costs  have  skyrock¬ 
eted — forcing  market-driven  demand 


responses  from  systems  of  systems. 

An  assumption  underlying  the  tech¬ 
niques  is  that  interoperability  issues  are — 
and  should  be — strongly  influenced  by  the 
need/desire  to  be  reactive  to  changing 
demands.  As  a  result,  these  modeling  and 
analysis  techniques  find  gaps  in  an  organi¬ 
zation’s  ability  to  react  to  changing 
demands.  The  goal  is  to  model  complex 
emergent  patterns  of  behavior  (and  gaps 
therein)  that  are  not  directly  intended  by 
any  single  governance  entity  within  a  com¬ 
plex  SoS.  The  techniques  model  physical 
and  social  aspects  of  enterprises  associat¬ 
ed  with  the  conception,  construction, 
fielding,  operations,  and  evolution  of 
complex  systems  and  systems  of  systems. 
An  enterprise  can  include  multiple  organi¬ 
zational  entities,  often  under  different 
management  and  ownership  structures. 
The  techniques  model  the  roles  and  inter¬ 
relationships  of  the  enterprise’s  physical 
and  social  elements  across  organizational 
entities  and  their  ability  to  form,  use,  and 
evolve  automated  systems  within  the  sys¬ 
tems’  operational  context  of  use — both  today 
and  for  the  future. 

The  techniques  probe  three  general 
categories  of  risk: 

1.  Performance.  The  risk  that  subsys¬ 
tems  within  system  elements  will  not 
interoperate  in  the  ways  needed  to 
respond  to  demand. 

2.  Composition.  The  risk  that  a  set  of 
systems  that  need  to  interoperate  with¬ 
in  a  given  SoS  cannot  be  made  to  inter¬ 
operate  in  the  ways  being  demanded  of 
them. 

3.  Mission.  The  risk  that  an  SoS  will  not 
function  within  its  operational  context 
of  use  in  the  ways  demanded  of  it. 

NATO  Interoperability  Probe 
Approach 

In  the  NATO  probe,  we  used  workshops 
and  small  group  interviews  to  examine 


Table  1 :  Perspectives  Represented  in  the  Workshops 


Client  Perspective 

Description 

Physical 

The  physical  realization  of  a  complex  system  or  SoS 
within  its  operational  context-of-use. 

Cognitive 

The  knowledge  associated  with  the  acquiring,  building,  or 
evolving  of  a  complex  system  or  SoS. 

Effects-based 

Mission  or  business  effects,  current  and  future,  that  the 
capabilities  provided  should  support. 
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interoperability  at  six  successively  broader 
levels  that  form  a  stratification: 

1.  Services,  systems,  and  know-how. 

2.  Activity  chains  involved  in  integrating 
components. 

3.  Activities  supporting  the  operational 
capability. 

4.  Orchestration  of  capabilities  by  a  crew 
and  operators. 

5.  Operational  performance  of  the  capa¬ 
bility. 

6.  Mission  environments. 

At  the  broadest  level — mission  envi¬ 
ronments — we  sought  interoperability 
risks  in  the  way  different  command 
authorities  were  able  to  collaborate.  At 
narrower  levels,  we  looked  at  the  way  dif¬ 
ferent  Command  and  Control  Infor¬ 
mation  Systems  assets  and  capabilities 
produced  combined  effects.  At  the  nar¬ 
rowest  levels,  we  examined  the  ability  of 
hardware,  software,  and  firmware  to  work 
together  as  effective  subsystems  within 
larger  systems. 

We  built  models  of  the  way  these  lev¬ 
els  interoperated  in  terms  of  the  relation¬ 
ships  between  people,  processes,  tech¬ 
nologies,  and  operational  demands  associ¬ 
ated  with  all  aspects  of  the  formation,  use, 
and  evolution  of  the  NATO  SoS.  The 
interoperability  models  were  produced  in 
stages  as  described  in  the  next  three  sec¬ 
tions: 

1.  Visual  Model  Representations. 

Layered,  graphical  depictions  that  con¬ 
formed  to  a  specific  syntax  of  symbols 
and  interconnection  rules.  This  stage 
was  interactive;  subject  matter  experts 
worked  with  the  modelers  in  a  work¬ 
shop  setting. 

2.  Model  Matrices.  A  set  of  stratified 
spreadsheets  that  juxtaposed  activities 
and  events  with  mission  environments. 
This  stage  was  generated  offline  by  the 
study  team  from  the  visual  model  rep¬ 
resentations  (Figure  1). 

3.  Interoperability  Landscapes.  The 
interrelationships  specified  in  the 
matrices.  This  stage  was  generated 
offline  and  became  the  primary  rea¬ 
soning  representation  back  to  the 
stakeholder  community  (Figures  2-5). 

Visual  Model  Representations 

Through  interviews  and  three  workshops 
(each  workshop  focused  on  one  of  the 
three  client  perspectives  listed  in  Table  1), 
the  SEI  team  and  the  client  representa¬ 
tives  created  models  to  ensure  that  the 
perspectives  of  all  relevant  stakeholders 
were  represented. 

To  initiate  the  modeling,  we  used  a 
brainstorming  aide,  referred  to  as 
Stakeholder  Collaboration  Analysis  or  the  4- 


Figure  1 :  Matrix  Stratification  With  Lxemplar  Entities 


Colors  for  short,  which  has  origins  in  war 
gaming.  It  facilitated  discussion  and  ini¬ 
tial  expression  of  the  dynamic  character¬ 
istics  of  the  social  and  technical  systems 
being  examined.  In  war  games,  blue  rep¬ 
resents  friendly  forces;  red,  the  enemy 
forces;  white,  the  referees;  and  black, 
intelligence.  We  modified  this  rubric  to  fit 
the  NATO  context. 


In  NATO’s  case,  we  applied  the  colors 
to  describe  the  program’s  capabilities 
(blue)  in  relation  to  the  particular 
demands  being  placed  upon  them  (red), 
within  the  context  of  what  is  driving  the 
mission  environment  (black).  White  was 
used  to  represent  the  management  of  the 
interoperability  among  all  of  these  con¬ 
stituents.  A  significant  study  team  hypoth- 


Figure  2:  Interoperability  Landscape 
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Figure  3:  Mission  Awareness  Landscape 
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Figure  4:  Orchestration  Landscape 


esis  was  that  NATO’s  focus  was  biased 
toward  managing  the  capabilities  (white/ 
blue)  in  a  way  that  was  divisive  to  the  ever- 
changing  demand  versus  the  mission  dri¬ 
ver  (red/black)  relationship.  Rotating 
through  the  color  quadrants ,  we  probed  for 
structural  (what,  how)  and  influential 
(who,  why)  aspects.  This  generated  a  wall 
full  of  sticky  notes  to  jump  start  the  visu¬ 
al  modeling. 

The  visual  models  were  constructed 
interactively  in  the  workshops  using 
Microsoft  Visiowith  a  custom  stencil  of 
symbols  and  rules  applied  to  assure  that 
only  the  allowed  symbols  and  appropriate 
connectors  were  used  within  and  between 
layers  of  the  model.  This  consolidated 
visual  model  representation  contained  five 
interlocking  layers: 

1.  Structure/Function.  The  physical 
structure  and  functioning  of  resources 
and  services. 

2.  Hierarchy.  The  formal  hierarchies 
and  standards  under  which  both  the 
non-digital  and  digital  aspects  of  the 
whole  are  held  accountable. 

3.  Trace.  The  digital  processes  and  soft¬ 
ware  that  interact  with  the  physical 
processes. 

4.  Demand.  The  organization  of  cus¬ 
tomers’  needs  as  demands  on  the  way 
the  enterprise  is  organized. 

5.  Synchronization.  The  lateral  relations 
of  synchronization  and  coordination 


within  the  enterprise  and  between  the 
enterprise  and  its  customers. 

In  between  client  sessions,  these  visu¬ 
al  model  entity-relationship  diagrams 
were  analyzed  for  patterns  (complex  or 
emergent3)  that  facilitate  the  structuring 
of  the  entities  according  to  the  stratifica- 

“The  patterns  revealed 
aligning  structures 
between  the 
mechanisms  that 
determined  the 
organization's  ability 
to  react  and 
manage  itself.” 

tion  (from  services  to  mission  environ¬ 
ments).  The  patterns  revealed  aligning 
structures  between  the  mechanisms  that 
determined  the  organization’s  ability  to 
react  and  manage  itself  (e.g.,  governance, 
actors,  design  authority — the  determining 
structures)  and  those  mechanisms  that 
carry  out  the  directives  of  the  determin¬ 
ing  structures  (e.g.,  systems,  processes. 


agents — the  determined  structures). 

Model  Matrices 

The  stage-one,  entity-relationship- style 
diagrams  rapidly  became  eye  charts ,  too 
complex  to  analyze  directly.  The  views 
conveyed  the  global  complexity  of  the  sit¬ 
uation,  providing  a  structural  snapshot  of 
the  dynamic  characteristics  of  this  com¬ 
plex  SoS.  However,  they  did  little  to  indi¬ 
cate  specific  interoperability  risks  among 
those  characteristics. 

In  the  probe’s  second  stage,  we  took 
advantage  of  the  defined  semantics  and 
rule-based  approach  of  the  diagramming 
technique  to  convert  the  diagrams  into  a 
stratified  matrix.  The  conversion  was  done 
using  an  automated  utility  that  leveraged 
recognizable  patterns  in  the  entity/con- 
nector  relationships  (e.g.,  a  hierarchical 
unit  that  controls  a  synchronization  of 
activities  produced  a  derived  entity  called 
an  orchestration).  The  conversion  layered 
the  connection  information  embedded  in 
the  diagram’s  semantics  from  the  point  of 
view  of  the  different  demands  being 
placed  on  the  systems;  see  [4]  for  details  of 
this  procedure. 

The  NATO  six-level  stratification  is 
illustrated  in  Figure  1.  The  core  levels — 
the  sub-matrices  labeled  1-6  (the  darker 
shaded  boxes) — form  a  value  stratifica¬ 
tion  that  progresses  from  low-level  ser¬ 
vices,  systems,  and  know-how  to  high- 
level  mission  environment  descriptions 
(this  progression  is  represented  by  the 
connecting  arrows  in  Figure  1).  The  other 
numbered  matrices  model  the  aligning 
structures  that  facilitate  the  overall  enter¬ 
prise’s  ability  to  react  to  the  mission  envi¬ 
ronment.  The  major  axes  are  composed 
of  the  simple  and  complex  objects4  that 
model  the  enterprise’s  assets,  capabilities, 
and  processes. 

Figure  1  also  includes  some  exemplars 
of  the  entity  types  that  populated  the  var¬ 
ious  sections  of  the  matrix,  such  as  mis¬ 
sion  situations,  drivers,  demand  situations, 
constituent  capabilities,  events,  processes 
(such  as  change  notification),  know-how, 
and  outcomes. 

Interoperability  Landscapes 

In  the  third  stage  of  the  probe,  we  ana¬ 
lyzed  the  stratified  matrix  and  produced 
interoperability  landscapes.  The  land¬ 
scapes  depict  the  connectedness  of  the  enti¬ 
ties,  sorted  so  that  neighboring  entities 
show  commonalities  and  differences  in 
their  degrees  of  connectedness.  An  inter¬ 
operability  landscape  (like  the  one  in 
Figure  2)  enabled  us  to  visualize  relation¬ 
ships  and  gaps  within  the  visual  model 
representations,  viewed  from  different 


Figure  5:  Performance  Risk  Landscape 
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Indirect  Management  Control 

The  slicing  process  can  be  customized  and  creatively  applied.  For  example,  when  the 
first-line  management  structures  were  suppressed,  the  impact  of  indirect  management 
control  jumped  out.  It  also  revealed  the  significant  separation  between  the  acquisition 
organization  and  the  line  command  structures.  This  figure  shows  the  vertical  command 
dependencies  in  this  SoS. 


perspectives  codified  by  the  matrix.  The 
columns  in  the  landscape  (Ron  Atkins’5 
simplicial  complexes)  are  organized  so  that 
entities  connected  through  a  shared  num¬ 
ber  of  interoperating  activities  are  next  to 
each  other  in  terms  of  their  height  and 
depth  dimensions.  The  height  dimension 
(q  in  the  landscape)  describes  the  number 
of  shared  underlying  activities;  the  higher 
the  q  between  columns,  the  more  related 
they  are.  The  depth  dimension  (k) 
describes  the  number  of  other  related 
columns  there  are  at  that  level  of  q.  For 
example,  the  landscape  in  Figure  2  shows 
peaks  separated  by  valleys.  These  valleys 
illustrate  the  gaps  between  the  different 
levels  of  shared  activity.  From  this  land¬ 
scape  and  its  underlying  matrices,  we 
gained  insight  into  which  17  relationships 
generate  the  high  peaks  and  which  10 
events  share  services  in  the  broad  plateau 
on  the  left  of  the  figure. 

Outcomes  About  NATO 
Interoperability  Risks  from 
the  Approach 

Using  the  modeling  and  analysis  tech¬ 
niques  approach  described  in  this  article, 
we  constructed  models  of  the  people, 
processes,  and  technologies  that  made  up 
the  NATO  modernization  program  and 
represented  the  way  demands  were  placed 
on  their  use.  Using  those  models  and  rep¬ 
resentations,  we  developed  an  objective 
view  that  reflected  the  major  interoper¬ 
ability  challenges  faced  by  the  program, 
using  interoperability  landscapes  to  dis¬ 
cover  and  illustrate  those  challenges.  We 
categorized  those  challenges  as  Type  III 
Mission  Risks,  Type  II  Composition  Risks, 
and  Type  I  Performance  Risks. 

Type  III  Mission  Risks 

Projecting6  from  Level  6  (mission  environ¬ 
ments)  in  the  matrix,  we  examined  how 
the  SoS  interoperates  within  its  demand- 
driven,  operational  context-of-use. 

Figure  3  is  the  three-dimensional 
depiction  of  our  mission  awareness  land¬ 
scape.  This  particular  example  shows  that 
the  predominant  mission  awareness  inte¬ 
gration  point  was  the  system  operator  and 
the  operator’s  display  console.  The  rest  of 
the  social  and  technical  systems  for  areas 
such  as  development,  support,  and  acqui¬ 
sition  were  virtually  unaware  of  mission- 
demand  complexity.  That  is,  these  systems 
did  not  interoperate  in  response  to 
demand  situations;  for  those  that  should 
have,  the  Type  III  risk  was  high7. 

Type  II  Composition  Risks 

Entering  the  matrix  at  Level  4,  the  orches¬ 


tration  level,  we  examined  whether  the 
systems  interoperated  in  the  ways  being 
demanded  of  them. 

After  ordering  and  ranking,  the  result¬ 
ing  orchestration  landscape  (see  Figure  4) 
revealed  obvious  islands  of  high  connec¬ 
tivity  with  broad  regions  of  separation. 
The  specific  entity  groupings  were  exam¬ 
ined  to  determine  if  the  separations  were 
warranted.  For  example,  gaps  revealed 
that  hardware  configuration  management 
was  quite  separate  and  poorly  orchestrated 
with  software  version  management.  The 
depth  of  the  valleys  indicates  that  the 
baseline  connective  tissue  (of  aspects  such  as 
change  management  and  revision  control) 
was  far  from  seamless  in  this  SoS. 

The  model  (at  the  modeled  fidelity)  is 
good  at  indicating  missing  connections;  it 
conversely  indicates  the  presence  of  con¬ 
nections  (peaks)  but  does  not  speak  to  the 
sufficiency  of  those  connections.  There¬ 
fore,  gaps  tend  to  be  truer  signs  of  inter¬ 
operability  risks  (because  it  is  hard  to  inter¬ 
operate  when  one  has  no  connection)  than 
peaks  are  guarantees  of  interoperability 
(because  high  connectivity  does  not  neces¬ 
sarily  mean  interoperability).  However, 
both  gaps  and  peaks  are  good  indicators  of 
worthy  areas  for  further  investigation. 

Type  I  Performance  Risks 

Entering  the  matrices  at  Level  3,  the  oper¬ 
ational  capability  level,  we  examined  how 
the  subsystems  within  system  elements 
were  or  were  not  connected. 

The  performance  risk  landscape 


(shown  in  Figure  5)  revealed  the  degree  of 
isolation  between  the  many  structural 
entities  in  this  SoS.  Once  again,  we  found 
a  high  likelihood  of  connectivity  gap-dri¬ 
ven  risks;  these  gaps  required  further 
examination  to  determine  the  severity  of 
consequences  before  declaring  specific 
risk  significance. 

In  our  three  categories,  we  identified 
interoperability  risks  and  visually  rein¬ 
forced  their  presence  by  the  landscape 
topologies.  The  objects  and  relationships 
depicted  in  the  landscapes  were  familiar  to 
the  client  and  served  to: 

•  Facilitate  constructive  dialogue  about 
mitigation  strategy. 

•  Justify  and  prioritize  follow-on  activi¬ 
ties,  such  as  detailed  impact  analysis, 
model  refinement  and  validation 
(through  detailed,  bottom-up  fact 
finding),  and  cost  analysis  in  targeted 
areas. 

Conclusions  About  the 
Modeling  and  Analysis 
Techniques 

The  examination  of  interoperability  is  a 
challenge  in  understanding  complexity. 
The  structural  models  produced  by  the 
techniques  bring  a  welcome  engineering 
rigor  to  the  process. 

In  part,  the  effectiveness  of  this  set  of 
model-based  analysis  techniques  can  be 
attributed  to  the  way  they  stress  the  need  to 
speak  in  the  client’s  language.  The  tech¬ 
nique  starts  with  client  artifacts  and  builds 
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visual  representations  (entity-relationship- 
style  diagrams)  that  are  understood  by  the 
client.  While  they  quickly  become  eye 
charts  that  are  too  complex  to  convey  any¬ 
thing  other  than  the  global  impression  of 
complexity,  these  diagrams  do  employ  rules 
of  object-connector  relationships  that 
facilitate  a  transformation  of  the  data  into 
a  stratified  matrix,  supporting  empirical 
analysis.  Overall,  the  techniques  produce  a 
rapid  (nominally  two  days  per  model)  snap¬ 
shot  of  interoperability  risks  from  the  per¬ 
spective  of  the  interviewed  stakeholders. 

Of  great  merit  in  the  techniques  is  the 
attention  paid  to  understanding  the  rela¬ 
tionship  of  the  operational  context  and 
the  supplied  technologies,  capabilities,  and 
governance  mechanisms8.  By  identifying 
gaps  in  their  alignment,  the  NATO  inter¬ 
operability  probe  team  identified  critical 
risks  that  are  often  overlooked.^ 
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Notes 

1.  The  techniques  used  were  a  demon¬ 
stration  of  Boxer  Research  Limited’s 
(BRL’s)  Projective  Analysis  (PAN). 
The  SEI  and  BRL  have  since  integrat¬ 
ed  uses  of  PAN  into  the  SEI  SoS 
Navigator  suite  of  products.  PAN  has 
been  applied  in  enterprises  in  such 
domains  as  manufacturing,  health 
care,  defense,  and  telecommunications 
[2].  For  example,  PAN  has  been  used 
in  support  of  Through  Life  Capability 
Management  practices  [3]  for  the 
United  Kingdom  Ministry  of  Defense. 
Projects  under  the  European  EURE¬ 
KA  program,  jointly  funded  with  the 
Department  of  Trade  and  Industry  in 
the  UK,  with  City  University  (London) 
as  a  subcontractor,  further  developed 
parts  of  the  technology. 

2.  This  article  has  been  drawn,  with  per¬ 
mission,  from  two  SEI  special  reports 
[4,  5]  and  from  information  about 
PAN  available  at  [6] .  We  have  used  fig¬ 
ures  utilized  in  our  NATO  research.  To 
protect  NATO’s  confidentiality,  some 
actual  specifics  have  been  removed: 
For  example,  the  vertical  marks  above 
simplical  complexes  in  the  landscape 
figures  each  represent  an  item  with  a 
name  that  has  been  removed. 


3.  The  patterns  are  represented  by 
model-generated  entities  emerging 
from  more  complex  interactions  than 
are  represented  by  simple  entity-con¬ 
nector-entity  constructs  (e.g.,  markets, 
orchestrations,  and  super- channels). 

4.  Tangible  assets,  such  as  control  mod¬ 
ules  or  design  expertise,  are  named  sim¬ 
ple  objects.  Patterns  of  relationships  that 
form  outcomes  or  mission  situations 
are  named  as  complex  objects. 

5.  The  form  of  description  behind  this 
matrix  format  uses  the  mathematics  of 
Ron  Atkins’  Q-analysis.  An  introduc¬ 
tion  to  this  can  be  found  in  [7].  The 
simplicial  complexes  are  derived  direct¬ 
ly  from  the  named  entities  in  the  visual 
model  and  from  the  patterns  of  objects 
generated  by  the  stratification  process. 

6.  Projection  is  the  systematic  process  used 
to  examine  the  matrix  representation 
of  the  modeled  SoS;  it  is  described  in 
detail  in  [4].  The  technique  uses  the 
stratification  to  provide  entry  points  at 
different  levels  of  complexity.  This 
provides  a  means  by  which  the  inter¬ 
operability  issues  can  be  decomposed 
at  different  levels  of  complexity. 

7.  If  the  consequence  of  the  detected 
condition  is  not  serious  (i.e.,  benign), 
the  risk  may  be  considered  low  [8] . 

8.  The  technique  models  the  structure¬ 
determining  processes  of  the  organi- 
zation-in- context  as  well  as  the  struc¬ 
ture-determined  processes  of  the  sys¬ 
tems  the  organization  uses. 
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Quality  and  Cost  -  It’s  Not  Either/Or: 

Making  the  Case  With  Cost  of  Quality 

George  Webb  LTC  Nanette  Patton 

45th  Range  Management  Squadron ,  Patrick  Mir  Force  Base  Ojfice  °f  the  Surgeon  General 

Today’s  organisations  must  be  committed  to  the  continuous  pursuit  of  quality  improvement  as  a  requirement  for  survival. 

Traditionally,  quality  and  cost  have  been  perceived  as  a  trade-off  decision.  Tor  this  reason,  the  main  purpose  and  benefit  of 
measuring  quality  costs  has  been  to  demonstrate  that  improved  quality  and  lower  costs  go  hand-in-hand.  Through  collection 
and  analysis  of  these  quality  costs,  improvement  is  translated  into  a  language  management  listens  to  and  responds  to:  money. 

This  article  provides  tools  and  techniques  to  help  infuse  cost  of  quality  (COQ)  concepts  into  the  project  team  activities  to  pro¬ 
mote  quality  improvement  throughout  the  full  project  life  cycle. 


There  have  been  many  changes  in  how 
DoD  project  management  applies 
quality  to  software  projects.  In  days  gone 
by,  there  was  a  separate  cost  for  quality 
attached  to  the  items  identified  within  an 
acquisition,  whether  for  software  or  for 
hardware.  As  acquisition,  project  manage¬ 
ment,  and  system  engineering  have 
evolved,  many  companies  have  replaced 
the  term  quality  with  other  terms,  such  as 
performance  and  best  practices.  Current  best 
practices  standard-bearers — such  as  the 
International  Organization  for  Standard¬ 
ization  (ISO),  the  Software  Engineering 
Institute’s  Capability  Maturity  Model® 
Integration  (CMMI®),  the  Project  Man¬ 
agement  Institute  (PMI),  and  the  IEEE — 
integrate  quality  into  everyone’s  work 
ethic  and  processes.  This  entails  creating 
integrated  product/process  teams  (IPTs) 
and  letting  the  individuals,  such  as  engi¬ 
neers,  logisticians,  configuration  man¬ 
agers,  testers,  and  project  mangers  (PMs), 
assume  responsibility  for  quality  within 
their  functional  processes. 

ISO  and  CMMI  argue  for  a  separate 
quality  group  to  maintain  objectivity. 
While  the  PM  has  the  final  responsibility 
for  quality,  a  quality  manager  can  be 
responsible  for  day-to-day  quality  by 
developing  and  implementing  a  quality 
effort.  Experience  has  shown,  however, 
that  when  funds  are  tight  and  time  is 
short,  the  quality  group  is  among  the  first 
cut  because  they  do  not  produce  a  physical 
product  for  the  customer.  Another  short¬ 
term  cost-saving  measure  is  for  the  PM  to 
select  someone  untrained  in  quality  assur¬ 
ance,  from  the  ranks  of  the  engineering 
staff,  to  be  the  quality  manager.  In  other 
cases,  there  may  be  no  identification  at  all 
of  a  separate  quality  process  owner  to 
oversee  this  critical  area;  consequently,  the 
COQ  remains  hidden  in  other  project 

®  The  Capability  Maturity  Model  and  CMMI  are  registered  in 
the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon 
University. 


costs.  When  quality  is  just  another  sub¬ 
process,  it  has  to  fight  for  attention,  prior¬ 
ity,  and  funding  like  all  the  others.  When 
this  occurs,  the  opportunity  to  identify 
and  correct  problems  early  in  the  life  cycle 
is  often  lost. 

This  maladaptive  application  of  quali¬ 
ty  principles  can  be  attributed  to  a  lack  of 
training,  management  support  for  quality 
processes,  and  adequate  quality  cost  mea¬ 
surement  systems.  It  is  more  likely  to 
occur  when  cost  and  schedule  become 
more  important  than  or  equal  to  quality. 

Figure  1  (taken  from  [1])  shows  the 
basic  cost  of  good  and  poor  quality. 

Evolution  in  the  Approach  to 
Quality 

In  decades  past,  the  focus  of  quality  was 
merely  finding  problems  at  the  end  of  an 
assembly  line  and  removing  the  defects 
before  shipping  to  the  consumer.  If  the 
product  did  not  meet  specifications,  it  was 
either  reworked  or  scrapped — both 
expensive  options.  This  approach  is  prone 
to  human  error  and  rarely  finds  all  defects. 
Furthermore,  this  quality  control  ap¬ 
proach  only  identified  the  defects  found 
through  a  random  sampling,  but  actually 
did  nothing  to  determine  the  root  cause  of 
the  problem  for  resolution. 

If  preventive  quality  measures  and 


rework  are  deferred  until  the  testing 
phase,  the  cost  of  change  is  40  to  100 
times  greater  than  if  the  defect  was  fixed 
when  it  was  created  [2].  The  testing  stage 
has  the  least  recovery  time  for  show-stopper 
problems  or  unexpectedly  large  amounts 
of  rework.  This  unpredictability  becomes 
a  large  contributing  factor  to  why  projects 
miss  their  schedules.  Furthermore,  this 
type  of  approach,  which  assumes  that 
doing  more  testing  leads  to  shipping  a  bet¬ 
ter  product,  only  works  up  to  a  point. 
With  pure  testing,  one  can  get  to  some¬ 
thing  approaching  5-Sigma  quality  (0.2 
defects  per  thousand  lines  of  code);  how¬ 
ever,  a  product  shipped  at  5-Sigma  is  per¬ 
ceived  as  inadequate  by  today’s  manufac¬ 
turing  standards  [3]. 

In  the  post- World  War  II  reconstruc¬ 
tion  years,  Dr.  W  Edwards  Deming  intro¬ 
duced  a  quality  program  that  simultane¬ 
ously  controlled  the  production  and  quali¬ 
ty  processes.  Unfortunately,  the  United 
States  did  not  adopt  these  principles  until 
the  1980s  with  the  introduction  of  the 
Total  Quality  Management  System.  Dem- 
ing’s  core  message — that  we  should  stop 
inspecting  defects  out  of  products  and 
start  building  quality  in — has  remained. 
The  common  thread  of  various  quality 
methodologies  is  that  the  project  team  will 
build  quality  into  the  system  design  and 
will  address  quality  continually  throughout 


Figure  1 :  Cost  of  Quality 
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the  life  cycle.  The  goal  is  to  identify  prob¬ 
lems  up  front  and  early,  allowing  corrective 
action  and  quality  prevention  to  take  place 
to  reduce  the  number  of  critical  defects 
found  at  the  end  of  the  assembly  line. 

This  goal  can  be  met  through  software 
quality  surveillance,  which  includes  walk¬ 
throughs,  peer  reviews,  inspections,  test¬ 
ing,  IPT  structures,  as  well  as  any  method 
that  identifies  quality  problems,  risks,  and 
operational  capability  weaknesses  as  early 
as  possible.  Approaching  quality  in  this 
manner  provides  early  corrective  action 
and  promotes  lower  quality  costs  upfront 
and  early,  thereby  reducing  end-of-pro- 
gram  cost  overruns  [4]. 

Today,  we  have  gone  a  step  further  by 
identifying  risks  which  may  have  the 
potential  to  change  engineering  require¬ 
ments,  operational  capabilities,  and  the 
quality  of  the  product.  In  the  spirit  of  risk 
management,  software  developers  can 
help  prevent  one  of  the  most  common 
causes  of  defects — ambiguous  require¬ 
ments — by  writing  comprehensive  accep¬ 
tance  tests  when  recording  each  require¬ 
ment.  Furthermore,  automating  these 
tests  and  running  them  as  part  of  frequent 
integration  builds  will  help  detect  defects 
when  they  happen. 

While  common  sense  says  that  pre¬ 
venting  defects  or  finding  them  when  they 
are  cheapest  to  fix  is  preferable  to  finding 
them  at  the  end  when  they  are  many  times 
more  expensive,  several  software  develop¬ 
ment  projects  fail  to  write  tests  upfront, 
do  inspections,  or  perform  frequent  inte¬ 
gration — despite  the  benefits. 

Why  We  Don’t  Implement 
Preventive  Quality  Processes 

Implementing  quality  processes  is  tedious, 
time-consuming  work  in  most  environ¬ 
ments.  And,  time  is  money.  There  is  docu¬ 
ment  inspection  (usually  several  hundred 
pages)  and  writing  early  tests  for  critical 
requirements  at  the  beginning  of  a  pro¬ 
ject.  It  is  hard  to  keep  the  tests  up-to-date 
as  the  requirements  change  and  even  hard¬ 
er  when  you  realize  that  you  have  to 
inspect  the  tests.  These  strategies  increase 
the  cost  of  implementing  quality  and  the 
return  on  investment  is  not  always  pre¬ 
dictable  with  a  high  degree  of  reliability, 
especially  when  the  requirements  and 
design  have  not  been  locked  in.  Thus,  it  is 
mind-wrenching  work  to  determine  which 
of  all  the  possible  strategies  for  imple¬ 
mentation  will  bring  the  best  value  to  the 
project. 

Carolyn  Fairbank,  CEO  of  the  Quality 
Assurance  Institute,  said: 

We’re  far  too  focused  on  product 


delivery,  not  process  capability. 
We’re  too  busy  trying  to  get  the 
product  out  the  door.  Granted,  this 
is  a  market-driven  phenomenon, 
but  we’ll  have  to  change  that  dead¬ 
line-driven  attitude  to  one  of  good 
processes.  If  you  get  the  process 
right,  the  product  will  have  a  far 
better  chance  at  success.  Unfortu¬ 
nately,  many  IT  professionals  still 
don’t  quite  understand  the  concept 
of  process  management.  [5] 

According  to  Karl  Wiegers: 

We  do  far  too  much  pretending  in 
software.  We  pretend  we  know 
who  our  users  are,  we  know  what 
their  needs  are,  that  we  won’t  have 
staff  turnover  problems,  that  we 
can  solve  all  technical  problems 
that  arise,  that  our  estimates  are 


“Today,  we  have  gone  a 
step  further  by 
identifying  risks  which 
may  have  the  potential 
to  change  engineering 
requirements,  operational 
capabilities, 
and  the  quality  of 
the  product.9* 


achievable,  and  that  nothing  unex¬ 
pected  will  happen.  Risk  manage¬ 
ment  is  about  discarding  the  rose- 
colored  glasses  and  confronting 
the  very  real  potential  of  undesir¬ 
able  events  conspiring  to  throw  our 
project  off  track.  [6] 


Risk  identification  requires  a  look  into 
the  future  as  to  the  potential  success  of 
the  program.  The  challenge  lies  in  the 
identification  of  risk  versus  current  prob¬ 
lems.  The  PMI  defines  risk  as  “an  uncer¬ 
tain  event  or  condition  that,  if  it  occurs, 
has  a  positive  or  negative  effect  on  a  pro¬ 
ject’s  objectives”  [7].  Current  problems  re¬ 
quire  attention  and  action  even  if  the 
immediate  remedy  is  to  defer  corrective 
action  until  later.  Risks  realized  may 
require  actions  that  lead  a  project  team  to 
proceed  in  a  different  direction  altogether 


or  canceling  the  project  entirely.  Project 
teams  must  accept  some  risks  due  to  other 
requirements,  conditions,  assumptions,  or 
constraints;  however,  if  a  project  team 
chooses  to  completely  ignore  risk,  they 
greatly  increase  the  probability  of  project 
failure. 

A  company’s  economic  status  or  con¬ 
dition  are  significant  factors  when  decid¬ 
ing  what  process  to  implement  to  track 
quality  cost  [8].  Pursglove  and  Dale  sug¬ 
gest  that  the  profitable  nature  of  the  busi¬ 
ness  can  make  it  more  difficult  to  con¬ 
vince  management  of  the  need  to  track 
COQ  [9].  For  example,  having  more  engi¬ 
neers  and  fewer  quality  assurance  people 
on  a  project  can  be  great  for  a  company’s 
short-term  financial  success.  However,  if 
project  staff  members  do  not  build  in 
quality  from  the  start,  a  greater  reliance  on 
product  rework  results.  The  organization 
will  eventually  pay  for  the  inadequate  qual¬ 
ity  as  customers  identify  problems  with 
the  product  or  service.  Engineering 
changes  must  take  place  before  the  cus¬ 
tomer  deems  the  product  usable.  These 
engineering  changes  late  in  the  develop¬ 
ment  process  may  result  in  a  product  or 
service  that  does  not  quite  meet  the  origi¬ 
nal  intent  on  the  capabilities  delivery;  this, 
in  turn,  can  lead  to  lost  business.  If  this 
happens  on  a  recurring  basis,  the  compa¬ 
ny  may  experience  competitive  and  finan¬ 
cial  difficulties.  If  so,  a  company  may  be 
more  open  to  performing  an  assessment 
in  an  attempt  to  get  back  on  track.  Then, 
after  collecting  and  analyzing  data  that 
reveals  a  quality  problem,  the  company 
finally  decides  to  track  quality  costs.  This 
may  also  be  the  time  when  the  company 
experiences  total  failure.  They  know 
something  has  to  be  done,  but  don’t  have 
a  well  thought-out  plan.  They  make  knee- 
jerk  decisions,  such  as  simply  canceling  the 
project  and  not  addressing  the  underlying 
quality  problems  in  their  processes;  that, 
in  turn,  causes  unintended  conflict  within 
the  organization. 

Not  having  a  clear  understanding  of 
the  actual  value  of  COQ  also  hinders  the 
adoption  of  quality  processes.  There  has 
been  a  persistent  misconception  in  the 
business  community  that  the  COQ  is  a 
cost  over  and  above  that  of  developing 
and  producing  a  project  to  meet  a  specific 
and  required  outcome  and  schedule.  The 
COQ,  regardless  if  it  is  software  or  hard¬ 
ware,  is  the  price  of  not  creating  a  quality 
product  or  service.  If  the  development 
process  was  perfect  with  no  problems  and 
there  was  no  possibility  of  substandard 
service,  failure  of  products,  or  defects  in 
their  manufacture,  then  organizations 
would  have  no  expenditures  on  COQ. 


24  CROSSTALK  The  Journal  of  Defense  Software  Engineering 


November  2008 


Quality  and  Cost  -  It’s  Not  Either/Or:  Making  the  Case  With  Cost  of  Quality 


Prevention 

Appraisal 

Represents  everything  a  company  spends  to  prevent  software 

Includes  the  money  spent  on  the  actual  testing  activity.  Any  and 

errors,  documentation  errors,  and  other  product-related  errors. 

all  activities  associated  with  searching  for  errors  in  the  software 

•  Staff  training 

(and  associated  product  materials)  fall  into  this  category. 

•  Requirements  analysis 

•  Design  reviews 

•  Early  prototyping 

•  Code  inspection 

•  Fault-tolerant  design 

•  Glass  box  testing 

•  Defensive  programming 

•  Black  box  testing 

•  Usability  analysis 

•  Beta  testing 

•  Clear  specifications 

•  Test  automation 

•  Accurate  internal  documentation 

•  Usability  testing 

•  Pre-purchase  evaluation  of  the  reliability  of  development  tools 

•  Pre-release  out-of-box  testing  by  customer  service  staff 

Internal  Failure 

External  Failure 

The  cost  of  coping  with  errors  discovered  during  development 

The  costs  of  coping  with  errors  discovered  after  the  product  is 

and  testing.  These  are  bugs  found  before  the  product  is 

released.  These  are  typically  errors  found  by  your  customers. 

released. 

•  Technical  support  calls 

•  Bug  fixes 

•  Answer  books  (for  support) 

•  Regression  testing 

•  Investigating  complaints 

•  Wasted  in-house  user  time 

•  Refunds  and  recalls 

•  Wasted  tester  time 

•  Interim  bug  fix  releases 

•  Wasted  writer  time 

•  Shipping  product  updates 

•  Wasted  marketer  time 

•  Warranty,  liability  costs 

•  Wasted  advertisements 

•  Public  relations  to  soften  bad  reviews 

•  Direct  cost  of  late  shipment 

•  Lost  sales 

•  Opportunity  cost  of  late  shipment 

•  Lost  customer  goodwill 

•  Supporting  multiple  versions  in  the  field 

•  Reseller  discounts  to  keep  them  selling  the  product 

Table  1 :  COQ  Data  Points 


COQ  is  the  sum  of  costs  incurred  in 
maintaining  acceptable  quality  levels  plus 
the  cost  of  failure  to  maintain  that  level 
(cost  of  poor  quality),  and  typically  ranges 
from  15-25  percent  of  total  cost  [10]. 

Philip  B.  Crosby’s  “Quality  Is  Free” 
concept  [11],  identified  two  main  cate¬ 
gories  of  quality  costs:  conformance  costs 
(cost  of  good  quality),  and  nonconfor¬ 
mance  costs  (cost  of  poor  quality). 

Conformance  costs  include  prevention 
and  appraisal  costs;  nonconformance 
costs  include  internal failures  as  well  as  exter¬ 
nal  failures  (Table  1,  from  [12]).  A  defect 
found  early  in  the  project  prior  to  cus¬ 
tomer  delivery  is  termed  an  internal  fail¬ 
ure.  A  defect  identified  after  the  product 
has  been  deployed  to  the  customer  is  an 
external  failure.  External  failures  can  also 
include  incompatibility  of  the  software 
with  legacy  software  installed  in  the  field, 
or  a  lack  of  commonality  between  redun¬ 
dant  systems. 

Beyond  not  clearly  understanding 
COQ  concepts,  key  decision  makers  in  an 
organization  may  lack  knowledge  in  deter¬ 
mining  quality  costs  and  the  principles  for 
collecting  quality  costs.  Without  knowing 
what  quality  principles  are,  an  individual 
or  organization  may  have  no  idea  where  to 
place  their  focus  to  obtain  quality  costs. 
The  organization  can  remedy  this  either 
by  ensuring  that  a  quality  curriculum  is 
included  in  the  training  for  project  staff 


and  senior  leadership  or  by  hiring  a  quali¬ 
ty  consultant  to  guide  the  organization. 

Getting  a  COQ  System  in 
Place 

Herb  Krasner  and  Dan  Houston  explain 
that  companies  need  to  answer  three  ques¬ 
tions  [13]: 

1 .  How  much  does  poor  software  quality 
cost? 

2.  How  much  does  good  software  quality 
cost? 

3.  How  good  is  our  software  quality? 
Once  these  questions  are  answered, 

the  project  team  can  compare  quality  costs 
to  overall  software  production  costs  and 
software  profits,  and  to  benchmarks  and 
norms.  They  can  also  better  analyze  prod¬ 
uct  quality  to  improve  their  competitive 
situation,  measure  improvement  actions 
and  the  bottom-line  effect  of  quality  pro¬ 
grams,  visibly  see  previously  hidden  costs 
related  to  poor  quality,  and  more  clearly 
see  the  economic  tradeoffs  involved  with 
software  quality. 

Even  if  the  project  team  cannot  mea¬ 
sure  all  of  these  costs  with  a  high  reliance, 
a  COQ  model  quantifies  (for  management 
and  executives)  the  amount  of  money 
being  lost  on  fixing  defects  and  delivering 
poor-quality  products.  This,  in  turn,  nega¬ 
tively  affects  their  bottom  line.  That  pro¬ 
vides  motivation  and  impetus  for  imple¬ 
menting  preventive  quality  measures. 


In  order  to  feed  decision  makers  those 
costs  with  some  degree  of  legitimacy,  a 
life-cycle  model  has  been  developed  to 
guide  this  work  (Figure  2,  next  page). 

Note:  If  no  data  source  exists  for  the 
collection  of  costs,  then  the  project  team 
will  have  to  use  some  type  of  analytical 
technique  to  develop  a  cost  estimate 
model.  With  this  model,  the  team  should 
be  able  to  establish  the  cost  estimates  for 
the  appropriate  quality  categories. 

Capturing  COQ 

This  step  in  the  life  cycle  is  twofold:  (1) 
identify  the  costs  and  (2)  determine  a 
method  for  entering  costs  into  an 
accounting  system  that  tracks  them 
throughout  system/ service  development. 

When  quality  costs  are  initially  deter¬ 
mined,  the  categories  included  are  the  vis¬ 
ible  ones. 

Oftentimes  it  is  what  you  do  not  know 
that  can  hurt  a  project.  Software  quality 
costs  are  not  always  easy  to  identify  with¬ 
in  programs.  Software  has  many  hidden 
costs  that  may  not  be  readily  apparent  to 
the  project  manager.  These  are  shown 
below  the  water  line  in  Figure  3  (next 
page,  adapted  from  [1]  and  [14])  and  in  the 
expanded  Figure  4  list  on  page  27. 

As  an  organization  internalizes  a 
broader  definition  of  poor  quality,  the 
hidden  portion  of  the  iceberg  becomes 
apparent.  Identifying  these  costs  opens  a 
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Figure  2:  The  Life  Cycle  of  COQ 


door  of  opportunity  and  the  project  team 
can  then  implement  processes  to  avoid 
more  costly  expenditures  later  in  the  pro¬ 
ject.  Furthermore,  the  team  increases  the 
probability  that  the  product/ service  will 


be  acceptable  to  the  customer  within  all 
required  elements  and  functions  that  the 
product/ service  is  supposed  to  deliver. 

A  proper  focus  on  quality  entails  iden¬ 
tifying  and  funding  quality  costs  and  insti¬ 


tutionalizing  quality  processes  at  the  very 
beginning.  It  also  requires  that  quality 
experts  supply  to  management  an  estimate 
of  the  total  quality  costs,  good  or  bad. 
Management  uses  the  information  ob¬ 
tained  through  the  quality  initiative  as  a 
tool  to  adjust  funding  allocations.  Once 
the  quality  experts  have  gathered  data, 
upper  management  can  determine  with  a 
clearer  picture  where  to  concentrate  qual¬ 
ity  efforts  and  funding  for  eventually 
achieving  a  greater  positive  impact  and 
promoting  a  successful  project. 

While  the  traditional  method  is  the  most 
used  method  in  collecting  costs  related  to 
quality  (according  to  the  American  Society 
for  Quality),  organizations  can  also  use  the 
defect  document  collection  method,  the 
time  and  attendance  collection  method,  or 
the  assessment  method.  The  PM  and  the 
quality  manager  (if  the  PM  has  designated 
one  for  the  project),  will  have  to  do  a  cost- 
benefit  analysis  to  determine  which 
method  is  the  best  fit  for  the  organization 
and  project,  given  available  resources. 

Feed  Total  COQ 

Having  identified  costs  and  a  method  for 
tracking  them,  diligent  data  collection  is 
now  required.  This  also  includes  incorpo¬ 
rating  previously  unidentified  costs 
revealed  during  ongoing  activities  that 
now  require  tracking. 

Review  the  Contract 

The  quality  group  must  be  conscious  of 
the  legal  terms  as  well  as  the  performance 
of  specific  tasks  within  any  contract  to 
support  product/ service  development 
and  delivery.  They  should  be  familiar  with 
specifics  to  ensure  the  contractor  is  com¬ 
pleting  required  tasks  throughout  the  life 
cycle  of  the  product.  There  may  be  spe¬ 
cific  tasks  that  occur  sporadically  during 
the  development  cycle  and  therefore 
require  a  more  concerted  follow-up. 

Review  the  Cost  Accounting 
Approach 

On  a  periodic  basis,  it  is  important  to 
ensure  that  the  cost  accounting  process 
and  repository  adequately  track  all  perti¬ 
nent  cost  information  collected  on  work 
currently  performed. 

The  system  may  require  refinement 
due  to  new  requirements  and / or  addition¬ 
al  costs  not  identified  at  the  start  of  the 
life  cycle.  This  may  also  require  changes  in 
the  data  collection  method. 

Review  Existing  Data  Sources 

It  is  also  important  to  ensure  that  the 
sources  used  for  cost  data  provide  the  best 


Figure  3:  The  Iceberg  Model  of  COQ 
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Excessive  overtime 
Pricing  or  billing  errors 
Premium  freight  costs 
Excessive  field  service  expenses 
Planning  delays 
Low  morale 

Excessive  employee  turnover 
Development  cost  of  failed 
product 

Overdue  receivables 
Excessive  system  costs 
Complaint  handling 
Expediting  costs 

Paperwork/documentation  is  late 
Customer  allowances 
Lack  of  follow-up  on  current 
programs  (require  re-testing) 
Excess  inventory 
Unused  capacity/scrap  cost 
Time  with  dissatisfied  customers 
Incorrectly  completed  sales  order 
Scheduling  delays 
Test  tool  certifications 
Training  of  operators 


Obtaining  user  licenses  for  off-the-shelf 
software 

Security  measures  for  information 
assurance 

Designated  Approving  Authority 
certifications 

Compatibility  issues  with  legacy 
Government  oversight  (for 
government  contracts) 

Risk  tracking  and  impacts  of 
external  dependencies 
Management  reserve  for  tracking  risks 
Training  in  risk  detection 
Possibility  for  developing  more  quality 
gates  in  development,  which  will  require 
additional  personnel  and  time. 

Bug  fixes 

Regression  testing 
Wasted  in-house  user  time 
Wasted  tester  time 
Wasted  writer  time 
Wasted  advertisements 
Direct  cost  of  late  shipment 
Opportunity  cost  of  late  shipment 


Figure  4:  More  Hidden  Quality  Cost  Data  Sets  Not  Easily  Identifiable 
library/content  /c070502a.asp>. 


available  data  in  terms  of  validity  and 
accuracy.  Do  not  hesitate  to  switch  to  a 
better  data  source  if  it  provides  data  that 
will  give  a  more  accurate  picture  of  where 
the  project  stands  at  a  particular  point 
within  the  process. 

Review  Costs  Proposed 

As  development  continues,  hidden  costs 
not  identified  at  the  start  will  reveal  them¬ 
selves.  Track  these  costs  along  with  those 
originally  proposed  to  facilitate  budget 
adjustments  and  to  recalculate  the  return 
on  investment  projections  in  order  more 
effectively  manage  expectations. 

Studies  (such  as  [9])  indicate  that  the 
further  along  in  the  process  quality  is 
worked  into  the  product  or  service  that 
the  higher  the  COQ  will  be.  The  project 
team  should  try  to  reduce  the  overall  cost 
of  each  product  or  service  by  establishing 
the  optimum  level  of  preventive  and 
appraisal  costs  that  minimizes  resultant 
error  costs.  The  net  result  of  quality 
improvement  should  be  a  reallocation  of 
costs  across  the  COQ  categories  resulting 
in  a  reduction  in  the  overall  COQ.  An 
example  of  this  [15]  is  shown  in  Figure  5. 

Apply  Professional  Judgment 

This  is  the  time  for  analysis  of  all  infor¬ 
mation  gathered  regarding  the  health  of 
the  program.  By  pushing  this  data  and 
analysis  to  the  appropriate  project  deci¬ 
sion  makers,  they  can  make  informed 
decisions  on  how  to  proceed  to  ensure 
project  success. 

Conclusion 

At  first  glance,  an  individual  might  be 
prone  to  think  that  collecting  quality  costs 
is  expensive,  adding  unnecessary  costs  to 
the  product  or  project.  Quality  is  not  free 
in  that  you  have  to  make  an  up-front 
investment  in  time,  money,  and  effort. 
However,  if  performed  properly  over  the 
full  life  cycle  of  the  project,  you  can 
recoup  the  resources  expended  for  quality 
processes  by  avoiding  rework  later  in  the 
project  life  cycle.  By  communicating  the 
quality  story  in  terms  of  dollars,  you  can 
enlist  the  help  of  senior  management  to 
infuse  quality  processes  throughout  the 
project  life  cycle  and  contain  project  costs 
for  the  long  haul.  As  with  many  project 
management  standards  and  guides  (e.g., 
[7]),  collecting  quality  costs  is  like  project 
planning;  it  is  cheaper  to  properly  plan 
than  it  is  to  plan  a  little  and  fail  a  lot.^ 
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Letters  to  the  Editor 


Dear  CROSSTALK  Editor, 

The  April  2008  article  by  Kym  Henderson  and  Dr.  Ofer  Zwikael — 
Does  Project  Performance  Stability  Exist ?  A  Re-examination  of  CPI  and 
Evaluation  of  SPI(t)  [Schedule  Performance  Index  (time)]  Stability — 
is  based  on  faulty  research.  While  serving  as  a  senior  program  ana¬ 
lyst  for  contract  performance  management  at  the  Office  of  the 
Secretary  of  Defense,  I  sponsored  research  on  Earned  Value 
Management  (EVM)  analysis  (by  Dr.  David  Christensen  and  his  Air 
Force  Institute  of  Technology  students)  and  provided  access  to  the 
defense  contract  database. 

Henderson  and  Zwikael  cite  “an  internal  DoD  [Naval  Air 
Systems  Command]  research  project”  by  Michael  Popp  and  state, 
“In  contrast  to  Christensen  and  associates  research,  which  used  data 
from  the  DAES  (Defense  Acquisition  Executive  System)  database, 
the  data  used  by  Popp  was  sourced  from  the  Contracts  Analysis 
System  database  ...”  They  are  mistaken:  DAES  and  CAS  are  the 
same  database  under  different  names,  now  part  of  the  Defense 
Acquisition  Management  Information  Retrieval  system. 

Building  on  earlier  work,  Christensen  and  Captain  Scott  Heise 
published  their  benchmark  paper  “Cost  Performance  Index  (CPI) 
Stability”  in  the  National  Contract  Management  journal.  They  analyzed 
data  from  155  contracts  that  met  DoD  EVM  requirements  and  dis¬ 
tinguished  between  contracts  having  stable  and  unstable  perfor¬ 
mance  measurement  baselines.  Henderson  and  Zwikael  do  not 
explain  how  Popp  selected  contracts  from  the  same  source. 

They  also  include  information  on  37  projects  from  Israel,  the 
United  Kingdom,  and  Australia.  I  suggest  that  these  disparate  pro¬ 
jects  did  not  implement  EVM  consistently,  as  on  DoD  contracts, 
and  that  the  analysis  lacks  rigor  (Israeli  data  were  analyzed  “using 
visual  inspection  of  charts”). 

Henderson  and  Zwikael  claim  to  have  “overturned  long¬ 
standing  findings  and  beliefs  on  CPI  stability.”  I  disagree. 
Christensen  and  Capt.  Heise  have  shown  that  CPI  stability  exists 
on  DoD  contracts.  Whether  that  “rule”  is  true  in  other  manage¬ 
ment  environments  is  uncertain,  but  the  article  does  not  make 
the  case  for  refuting  it. 

— Wayne  F.  Abba 
<abbaconsulting@cox.net> 


Mr.  Popp’s  Naval  Air  Systems  Command  research  provided  a 
unique  independent  view  of  the  applicability  of  the  rule  utilizing 
data  now  known,  thanks  to  Abba’s  correction,  to  also  come  from 
the  same  restricted  data  source  utilized  by  Christensen. 

Popp’s  report  corroborates  the  research  using  commercial  sec¬ 
tor  EVM  data  and  practitioner  observations  that  the  CPI  stability 
rule  does  not  always  apply  on  their  projects.  Abba’s  letter  provides 
no  basis  for  modifying  the  paper’s  conclusions. 

The  Popp  report  and  Christensen  research  highlight  many  occa¬ 
sions  for  DoD  projects  where  the  rule  applies.  Follow-on  research 
was  recommended  to  ascertain  any  characteristics  which  result  in 
early  CPI  stability  and  also  where  a  progressively  improving  CPI 
precluded  stability. 

This,  with  Earned  Schedule  method  research,  provides  oppor¬ 
tunities  for  enhancing  project  management  practice,  project  perfor¬ 
mance,  and  the  application  of  EVM.  We  encourage  more  studies  in 
various  project  environments  focusing  on  this  important — still  to 
be  fully  solved — area. 

— Kym  Henderson  — Dr.  Ofer  Zwikael 

<  Kym .  Hender  s  on@froggy.  com  >  <ofer.zwikael@vuw.ac.nz> 


Dear  CROSSTALK  Editor, 

In  your  June  2008  edition,  you  asked  for  stories  on  “how  we  have 
helped.”  Well,  here  goes:  CROSSTALK  is  considered  our  base 
resource  for  the  two  Software  Acquisition  Management  courses 
here  at  the  Defense  Acquisition  University.  Almost  every  lesson 
relies  on  both  past  and  present  CROSSTALK  articles.  Many 
lessons  learned  and  best  practice  stories  have  been  extracted  from 
CROSSTALK  and  used  as  anecdotes  in  our  classes.  Students  and 
instructors  tell  us  that  they  use  it  to  find  out  the  latest  happenings 
in  the  DoD  software  industry.  It  is  an  invaluable  tool  that  allows 
DoD  software  professionals  to  keep  up  with  the  fast  pace  of  tech¬ 
nology.  The  fact  that  software  is  taking  over  the  functionality  in  all 
of  our  major  systems  makes  CROSSTALK  extremely  important 
to  the  DoD  software  acquisition  professional! 


— Bob  Skertic 
<Bob.Skertic@dau.mil> 


Dear  CROSSTALK  Editor, 

We  would  like  to  respond  to  Mr.  Abba’s  letter. 

The  article’s  purpose  was  to  re-examine  the  CPI  stability  rule 
(the  rule)  to  see  whether  the  Earned  Schedule  SPI(t)  exhibited  con¬ 
sistent  stability  behavior.  The  rule  has  been  claimed  in  various 
authoritative  sources  to  have  unqualified  universal  applicability  for, 
as  we  stated  in  the  article,  “all  projects  using  the  EVM  [Earned 
Value  Management]  method.” 

The  article’s  conclusion  that  the  referenced  claims  of  unquali¬ 
fied  universal  applicability  of  the  rule  cannot  be  generalized  as  self- 
evident.  Notably,  Abba’s  statement  of  uncertainty  on  whether  the 
rule  “is  true  in  other  management  environments”  significantly 
revises  the  previous  unqualified  claims  of  its  universal  applicability. 

Abba’s  letter  illustrates  that  the  rule  has  been  observed  by  one 
principal  researcher  using  data  (with  one  exception)  from  the  DoD 
DAES  database  to  which  access  for  research  is  restricted. 
Christensen’s  research  has,  to  the  best  of  our  knowledge,  not  been 
independently  corroborated  by  other  researchers,  an  important  pre¬ 
requisite  for  claiming  general  applicability. 


Dear  CROSSTALK  Editor, 

While  I  agree  that  some  improvement  is  better  than  no  improve¬ 
ment,  I  don’t  agree  with  the  idea — from  Quality  Processes  Yield  Quality 
Products  (June  2008)  by  Thomas  D.  Neff — that  any  process 
improvement  model  can  be  used  by  any  organization.  I  also  don’t 
agree  with  Neff’s  ideas  to  “just  do  it,”  and  that  “it  doesn’t  matter” 
which  model  you  choose. 

Over  the  last  25  years,  I  have  used,  taught,  and  consulted  on 
several  management  models,  each  having  assumptions  and  values 
built  into  them.  By  the  same  token,  each  client  organization  has 
taken- for-granted  assumptions  and  values  built  into  its  culture. 

To  achieve  maximum  bang  for  the  buck,  organizations  are  well- 
advised  to  take  a  few  minutes  to  figure  out  who  they  are  before  they 
select  a  process  improvement  model.  With  that  self-knowledge  in¬ 
hand,  an  organization  can  seek  out  a  model  that  best  fits  its  culture. 
The  closer  the  match,  the  better  perceived  outcomes  will  be. 

— Gaylord  Reagan 
<greagan@attglobal.net> 
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BackTalk 


Clouds  From  Both  Sides,  Now 


I  have  a  strange  quirk,  one  of  several.  When  I  read  sophistic 
technical,  political,  or  commercial  articles,  I  often  associate 
them  with  a  song  stowed  in  my  cerebral  archive.  More  accurate¬ 
ly,  I  start  hearing  the  song  crescendo  in  my  head  to  the  point  I 
can’t  concentrate  on  the  article.  Such  is  the  case  with  the  latest 
flood  of  cloud  computing  articles. 

With  big  guns  like  Google,  Microsoft,  Sun,  IBM,  Dell,  and 
Amazon  seeding  cloud  computing  technology,  the  forecast  is 
partly  cloudy  to  overcast  with  an  eventual  downpour. 

Every  time  I  try  to  get  my  head  into  cloud  computing,  I  hear 
Greg  Lake’s  carnival  barker  voice  from  the  Emerson,  Lake  & 
Palmer  song  “Karn  Evil  9.” 

Welcome  back  my  friends 
To  the  show  that  never  ends 
Were  so  glad  you  could  attend 
Come  inside,  come  inside!  [1] 

I’m  sure  you  can  help  me  see  the  light  ...  or  cloud  the  light? 
I’m  not  sure.  Distributed  computing?  Not  a  problem.  Computing 
Grid?  Makes  sense  for  large  scale  operations.  Software-as-a- 
Service?  Emphasize  service,  and  I’m  on  board.  However,  cloud 
computing  and  its  everything-as-a-service  mantra  sounds  more  like 
mountebank  bellows  than  a  reasoned  thesis. 

Part  the  cloud  jargon  and  I  see  computing  rental,  leasing,  and 
sharing.  You  basically  rent  and  share  processing  time,  data  stor¬ 
age,  security,  platforms,  applications,  and  so  forth.  I  know  it’s  so 
much  more  than  that,  but  is  it? 

You  don’t  own  it,  you  can’t  control  it,  and  it  has  no  intrinsic 
resale  or  depreciation  value.  So  why  not  use  rental,  lease,  or  time- 
share  nomenclature?  Not  sexy  enough,  too  transparent,  or  is  it 
burdened  with  negative  connotations  of  rental  and  timeshare 
property?  “Yes  you,  Joe  Shmuck,  can  have  your  own  slice  (for 
two  weeks)  of  paradise  anytime  you  want  (except  weekends,  hol¬ 
idays,  on  sunny  days,  and  during  picturesque  sunsets).” 

What  are  the  anticipated  advantages  of  cloud  computing? 
Topping  the  list  is  the  reduction  in  IT  capital  expenditures.  No 
servers,  computers,  applications,  storage,  or  personnel  to  main¬ 
tain  them.  Well,  you  will  likely  need  some  type  of  computer  to 
connect  to  the  cloud  and  support  to  maintain  and  configure  your 
computers  to  assure  cloud  compatibility. 

Where  do  those  costs  go?  The  cloud  service  provider  (CSP) 
picks  them  up  and  passes  them  back  to  you  in  a  little-extra-fee- 
for-service.  Hey,  silver  linings  are  not  free!  Of  course,  your 
monthly  cloud  bill  will  be  as  clear  as  your  cable,  phone,  or 
favorite  utility  bill. 

How  about  device  independence?  Do  you  really  think  I’ll  be 
able  to  use  my  iPhone  to  access  any  cloud?  “Sorry  sir,  iPhones 
only  work  with  iClouds.  For  cloud  nine  you  will  need  the  new 
Nimbostratus  from  Micro-cloud.” 

What  about  access  to  supercomputing  power  and  flexibility? 
You  want  supercomputing?  “Miss,  supercomputing  is  for  Silver 
Cloud  Members  only.  All  -ilities  have  their  own  surcharge  and  you 
will  be  required  to  purchase  a  cloud  security  undercoating.” 

What  about  the  efficiencies  of  centralization?  You  mean 
multi-tenancy?  Sure  six  guys  can  pool  resources  to  rent  an  apart¬ 
ment  easier  than  one,  but  that  does  not  guarantee  optimal  condi¬ 
tions.  When  everyone  wants  to  run  payroll,  invoices,  or  crunch 
out  the  human  genome  at  the  same  time — not  that  it  would  ever 
happen — how  efficient  can  that  be?  There  is  a  reason  the 


American  Dream  involves  single-tenancy,  independence,  and 
self-determination.  For  me.  I’ll  stick  with  my  own  little  cumulus 
humilis  (a.k.a.  a  fair  weather  cloud)  and  compute  along.  And,  if 
by  chance,  we  meet  on  the  super  jet  stream,  remember:  I’m  old 
school — hey  you,  get  off  of  my  cloud! 

I  love  the  argument,  “you  don’t  generate  your  own  electricity 
so  why  generate  your  own  computing?”  My  short  answer: 
because  I  can.  My  long  answer:  because  electricity  is  a  utility  and 
I’m  not  ready  to  concede  that  the  tools  I  use  to  create,  design, 
produce,  and  improve  are  mere  utilities.  When  I  tinker  with, 
tweak,  and  maintain  my  computer,  I  discover  new  ways  to  use  it 
and  open  new  doors.  Would  you  ask  a  Jedi  to  give  up  his  light 
saber  to  timeshare  a  cloud  saber? 

How  about  interoperability?  Let’s  see,  there  are  cloud  appli¬ 
cations,  cloud  services,  cloud  platforms,  cloud  storage,  and  com¬ 
peting  CSPs.  How  will  these  be  interoperable,  or,  “how  do  you 
catch  a  cloud  and  pin  it  down?”  [2]  Wait,  I  know!  The  high  cap¬ 
ital  costs  to  start  a  CSP  will  provide  a  barrier  to  the  CSP  market 
and  drive  the  industry  to  an  oligarchy,  monopoly  or  ...  gasp  ...  a 
Googleopoly.  Then  you  will  have  interoperability  via  collusion, 
soon  followed  by  National  Cloud  Care,  the  Cloud  Protection 
Agency,  Cloud  Footprints  requiring  Cloud  Offsets,  and  No 
Cloud  Left  Behind. 

You  can’t  beat  the  reliability  of  cloud  computing’s  multiple 
servers,  sites,  and  Continuity  of  Operations  Plans  (COOP).  Oh 
yes,  the  ever-present  Cloud  COOP.  Well,  if  the  reliability  and  ser¬ 
vice  of  present-day  ISPs  are  any  indication  of  the  reliability  and 
service  of  future  CSPs,  then  I  suggest  hip  waders.  “Sir,  I’m  look¬ 
ing  at  our  cloud  and  it  is  up  and  fully  functional.  The  problem 
must  be  with  your  thin  client.  Unfortunately,  our  cloud  coverage 
does  not  extend  to  thin,  lean,  or  slim  clients.” 

Ask  Amazon’s  S3  clients  (July  2008)  or  the  London  Stock 
Exchange  (September  2008)  about  reliability  and  the  effects  of 
software  as  a  non- service.  But  what’s  a  million- dollar  trade  com¬ 
mission  loss  amongst  cloud  members?  Besides,  that  was  not  real 
cloud  computing,  just  contrails. 

Listen,  can  you  hear  Joni  Mitchell?  She’s  a  true  cloud  afi¬ 
cionado: 

I’ve  looked  at  clouds  from  both  sides  now 
From  up  and  down,  and  still  somehow 
It’s  cloud  illusions  I  recall 
I  really  don’t  know  clouds  at  all.  [3] 

Don’t  let  me  rain  on  your  parade  or  cloud  your  judgment. 
You  may  find  a  silver  lining  in  them  thar  computer  clouds.  I’ve 
been  wrong  before  and  I’m  sure  I’ll  be  wrong  again.  If  you  must 
...  send  in  the  clouds  ...  there  ought  to  be  clouds  ...  well,  maybe 
next  year. 

— Gary  A.  Petersen 

Arrowpoint  Solutions,  Inc. 
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